Search Results: "ranty"

13 July 2020

Antoine Beaupr : Not recommending Purism

This is just a quick note to mention that I have updated my hardware documentation on the Librem 13v4 laptop. It has unfortunately turned into a rather lengthy (and ranty) piece about Purism. Let's just say that waiting weeks for your replacement laptop (yes, it died again) does wonders for creativity. To quote the full review:

TL;DR: I recommend people avoid the Purism brand and products. I find they have questionable politics, operate in a "libre-washing" fashion, and produce unreliable hardware. Will not buy again.

People who have read the article might want to jump directly to the new sections:

I have also added the minor section of the missing mic jack. I realize that some folks (particularly at Debian) might still work at Purism, and that this article might be demoralizing for their work. If that is the case, I am sorry this article triggered you in any way and I hope this can act as a disclaimer. But I feel it is my duty to document the issues I am going through, as a user, and to call bullshit when I see it (let's face it, the anti-interdiction stuff and the Purism 5 crowd-funding campaign were total bullshit). I also understand that the pandemic makes life hard for everyone, and probably makes a bad situation at Purism worse. But those problems existed before the pandemic happened. They were issues I had identified in 2019 and that I simply never got around to document. I wish that people wishing to support the free software movement would spend their energy towards organisations that actually do honest work in that direction, like System76 and Pine64. And if you're going to go crazy with an experimental free hardware design, why not go retro with the MNT Reform. In the meantime, if you're looking for a phone, I recommend you give the Fairphone a fair chance. It really is a "fair" (as in, not the best, but okay) phone that you can moderately liberate, and it actually frigging works. See also my hardware review of the FP2. Update: this kind of blew up, for my standards: 10k visitors in ~24h while I usually get about 1k visitors after a week on any regular blog post. There were more discussions on the subject here:

Trigger warning: some of those threads include personal insults and explicitly venture into the free speech discussion, with predictable (sad) consequences...

9 July 2020

Enrico Zini: Laptop migration

My laptop battery started to explode in slow motion. HP requires 10 business days to repair my laptop under warranty, and I cannot afford that length of downtime. Alternatively, HP quoted me 375 + VAT for on-site repairs, which I tought was very funny. For 376.55 + VAT, which is pretty much exactly the same amount, I bought instead a refurbished ThinkPad X240 with a dual-core I5, 8G of RAM, 250G SSD, and a 1920x1080 IPS display, to use as a spare while my laptop is being repaired. I'd like to thank HP for giving me the opportunity to own a ThinkPad. Since I'm migrating all my system to the spare and then (hopefully) back, I'm documenting what I need to be fully productive on new hardware. Install Debian A basic Debian netinst with no tasks selected is good enough to get going. Note that if wifi worked in Debian Installer, it doesn't mean that it will work in the minimal system it installed. See here for instructions on quickly bringing up wifi on a newly installed minimal system. Copy /home A simple tar of /home is all I needed to copy my data over. A neat way to do it was connecting the two laptops with an ethernet cable, and using netcat:

# On the source
tar -C / -zcf - home   nc -l -p 12345 -N
# On the target
nc 10.0.0.1 12345   tar -C / -zxf -

Since the data travel unencrypted in this way, don't do it over wifi. Install packages I maintain a few simple local metapackages that depend on the packages I usually used. I could just install those and let apt bring in their dependencies. For the build dependencies of the programs I develop, I use mk-build-deps from the devscripts package to create metapackages that make sure they are installed. Here's an extract from debian/control of the metapackage:

Source: enrico
Section: admin
Priority: optional
Maintainer: Enrico Zini <enrico@debian.org>
Build-Depends: debhelper (>= 11)
Standards-Version: 3.7.2.1
Package: enrico
Section: admin
Architecture: all
Depends:
  mc, mmv, moreutils, powertop, syncmaildir, notmuch,
  ncdu, vcsh, ddate, jq, git-annex, eatmydata,
  vdirsyncer, khal, etckeeper, moc, pwgen
Description: Enrico's working environment
Package: enrico-devel
Section: devel
Architecture: all
Depends:
  git, python3-git, git-svn, gitk, ansible, fabric,
  valgrind, kcachegrind, zeal, meld, d-feet, flake8, mypy, ipython3,
  strace, ltrace
Description: Enrico's development environment
Package: enrico-gui
Section: x11
Architecture: all
Depends:
  xclip, gnome-terminal, qalculate-gtk, liferea, gajim,
  mumble, sm, syncthing, virt-manager
Recommends: k3b
Description: Enrico's GUI environment
Package: enrico-sanity
Section: admin
Architecture: all
Conflicts: libapache2-mod-php, libapache2-mod-php5, php5, php5-cgi, php5-fpm, libapache2-mod-php7.0, php7.0, libphp7.0-embed, libphp-embed, libphp5-embed
Description: Enrico's sanity
 Metapackage with a list of packages that I do not want anywhere near my
 system.

System-wide customizations I tend to avoid changing system-wide configuration as much as possible, so copying over /home and installing packages takes care of 99% of my needs. There are a few system-wide tweaks I cannot do without:

setup postfix to send mail using my mail server
copy Network Manager system connections files in /etc/NetworkManager/system-connections/
update-alternatives --config editor

For postfix, I have a little ansible playbook that takes care of it. Network Manager system connections need to be copied manually: a plain copy and a systemctl restart network-manager are enough. Note that Network Manager will ignore the files unless their owner and permissions are what it expects. Fine tuning Comparing the output of dpkg --get-selections between the old and the new system might highlight packages manually installed in a hurry and not added to the metapackages. Finally, what remains is fixing the sad state of mimetype associations, which seem to associate opening file depending on whatever application was installed last, phases of the moon, and what option is the most annoying. Currently on my system, PDFs are opened in inkscape by xdg-open and in calibre by run-mailcap. Let's see how long it takes to figure this one out.

10 May 2020

Russell Coker: IT Asset Management

In my last full-time position I managed the asset tracking database for my employer. It was one of those things that someone needed to do, and it seemed that only way that someone wouldn t equate to no-one was for me to do it which was ok. We used Snipe IT [1] to track the assets. I don t have enough experience with asset tracking to say that Snipe is better or worse than average, but it basically did the job. Asset serial numbers are stored, you can have asset types that allow you to just add one more of the particular item, purchase dates are stored which makes warranty tracking easier, and every asset is associated with a person or listed as available. While I can t say that Snipe IT is better than other products I can say that it will do the job reasonably well. One problem that I didn t discover until way too late was the fact that the finance people weren t tracking serial numbers and that some assets in the database had the same asset IDs as the finance department and some had different ones. The best advice I can give to anyone who gets involved with asset tracking is to immediately chat to finance about how they track things, you need to know if the same asset IDs are used and if serial numbers are tracked by finance. I was pleased to discover that my colleagues were all honourable people as there was no apparent evaporation of valuable assets even though there was little ability to discover who might have been the last person to use some of the assets. One problem that I ve seen at many places is treating small items like keyboards and mice as assets . I think that anything that is worth less than 1 hour s pay at the minimum wage (the price of a typical PC keyboard or mouse) isn t worth tracking, treat it as a disposable item. If you hire a programmer who requests an unusually expensive keyboard or mouse (as some do) it still won t be a lot of money when compared to their salary. Some of the older keyboards and mice that companies have are nasty, months of people eating lunch over them leaves them greasy and sticky. I think that the best thing to do with the keyboards and mice is to give them away when people leave and when new people join the company buy new hardware for them. If a company can t spend $25 on a new keyboard and mouse for each new employee then they either have a massive problem of staff turnover or a lack of priority on morale.

[1] https://snipeitapp.com/

Norbert Preining: Updating Dovecot for Debian

A tweet of a friend pointed me at the removal of dovecot from Debian/testing, which surprised me a bit. Investigating the situation it seems that Dovecot in Debian is lagging a bit behind in releases, and hasn t seen responses to some RC bugs. This sounds critical to me as dovecot is a core part of many mail setups, so I prepared updated packages.

Based on the latest released version of Dovecot, 2.3.10, I have made a package starting from the current Debian packaging and adjusted to the newer upstream. The package builds on Debian Buster (10), Testing, and Unstable on i386 and x64 archs. The packages are available on OBS, as usual: For Unstable:

deb https://download.opensuse.org/repositories/home:/npreining:/debian-dovecot/Debian_Unstable/ ./

For Testing:

deb https://download.opensuse.org/repositories/home:/npreining:/debian-dovecot/Debian_Testing/ ./

For Debian 10 Buster:

deb https://download.opensuse.org/repositories/home:/npreining:/debian-dovecot/Debian_10/ ./

To make these repositories work, don t forget that you need to import my OBS gpg key: obs-npreining.asc, best to download it and put the file into /etc/apt/trusted.gpg.d/obs-npreining.asc. These packages are provided without any warranty. Enjoy.

24 March 2020

Norbert Preining: KDE/Plasma 5.18 for Debian

Update 2020-04-03: Please see this post for updated location of the packages!!! Now for i586 and amd64 architectures! I have been trying out the Plasma Desktop for one week now, and I am very positively surprised. Compared to the clumsy history of KDE3, the current desktop is extremely small-footprint and smooth, surprisingly. Integration is as expected great, and mixing programs from the other world (Gtk/Gnome) works also extremely smooth.

If there are a few things I would change, then mostly the chaos about kwallet and Gnome keyring. I would love to have one secret storage, and it seems that Gnome Keyring is preferrable, but this is in flux at the moment. Also, it is not that pressing, because I have moved all my passwords into pass and thus don t need the secret storage that much anymore. So, after a bit of working with Plasma, I realized that Debian still ships an old version, the most recent being 5.18.3 LTS. Thus, I embarked onto a journey of updating all the necessary packages, and there are a lot: in total 106 packages I did update (and one new one!) until I finally had a new plasma-desktop package available. If you are interested, there are binaries for amd64 and sources in my Debian repository (WARNING: These packages are for Debian/sid and maybe testing, and cannot be used with Buster!):

deb https://www.preining.info/debian unstable kde
deb-src https://www.preining.info/debian unstable kde

As usual, don t forget to import my GPG key, and all packages are without warranty :-; There are two packages that I didn t manage to update: kde-gtk-config which has changed a lot and contains far less files, and breeze-icons which fails on its own symlink tests. If anyone has an idea, please let me know. If other packages are missing, please also drop me a line and I ll try to update them. Enjoy.

18 October 2017

Norbert Preining: Kobo firmware 4.6.9995 mega update (KSM, nickel patch, ssh, fonts)

It has been ages that I haven t updated the MegaUpdate package for Kobo. Now that a new and seemingly rather bug-free and quick firmware release (4.6.9995) has been released, I finally took the time to update the whole package to the latest releases of all the included items. The update includes all my favorite patches and features: Kobo Start Menu, koreader, coolreader, pbchess, ssh access, custom dictionaries, and some side-loaded fonts.

So what are all these items:

firmware (thread): the basic software of the device, shipped by Kobo company
Metazoa firmware patches (thread): fix some layout options and functionalities, see below for details.
Kobo Start Menu (V08, update 5b thread): a menu that pops up before the reading software (nickel) starts, which allows to start alternative readers (like koreader) etc.
KOreader (koreader-nightly-20171004, thread): an alternative document reader that supports epub, azw, pdf, djvu and many more
pbchess and CoolReader (2017.10.14, thread): a chess program and another alternative reader, bundled together with several other games
kobohack (web site): I only use the ssh server
ssh access (old post: makes a full computer from your device by allowing you to log into it via ssh
custom dictionaries (thread): this fix updates dictionaries from the folder customdicts to the Kobo dictionary folder. For creating your own Japanese-English dictionary, see this blog entry
side-loaded fonts: GentiumBasic and GentiumBookBasic, Verdana, DroidSerif, and Charter-eInk

Install procedure Download Mark6 Kobo GloHD firmware: Kobo 4.6.9995 for GloHD Mega update: Kobo-4.6.9995-combined/Mark6/KoboRoot.tgz Mark5 Aura firmware: Kobo 4.6.9995 for Aura Mega update: Kobo-4.6.9995-combined/Mark5/KoboRoot.tgz Mark4 Kobo Glo, Aura HD firmware: Kobo 4.6.9995 for Glo and AuraHD Mega update: Kobo-4.6.9995-combined/Mark4/KoboRoot.tgz Latest firmware Warning: Sideloading or crossloading the incorrect firmware can break/brick your device. The link below is for Kobo GloHD ONLY. The first step is to update the Kobo to the latest firmware. This can easily be done by just getting the latest firmware from the links above and unpacking the zip file into the .kobo directory on your device. Eject and enjoy the updating procedure. Mega update Get the combined KoboRoot.tgz for your device from the links above and put it into the .kobo directory, then eject and enjoy the updating procedure again. After this the device should reboot and you will be kicked into KSM, from where after some time of waiting Nickel will be started. If you consider the fonts too small, select Configure, then the General, and add item, then select kobomenuFontsize=55 and save. Remarks to some of the items included The full list of included things is above, here are only some notes about what specific I have done.

Metazoa firmware patches Included patches from the Metazoa firmware patches:

Custom left & right margins
Fix three KePub fullScreenReading bugs
Change dicthtml strings to micthtml
Default ePub monospace font (Courier)
Custom reading footer style
Dictionary pop-up frame size increase
Increase The Cover Size In Library
Increasing The View Details Container
New home screen increasing cover size
Reading stats/Author name cut when the series is showing bug fix 
New home screen subtitle custom font
Custom font to Collection and Authors names

If you need/want different patches, you need to do the patching yourself.

kobohack-h Kobohack (latest version 20150110) originally provided updated libraries and optimizations, but unfortunately it is now completely outdated and using it is not recommended for the library part. I only include the ssh server (dropbear) so that connections to the Kobo via ssh.
ssh fixes See the detailed instructions here, the necessary files are already included in the mega upload. It updates the /etc/inittab to run also /etc/init.d/rcS2, and this one again starts the inetd server and run user supplied commands in /mnt/onboard/run.sh which is where your documents are.
Custom dictionaries The necessary directories and scripts are already included in the above combined KoboRoot.tgz, so nothing to be done but dropping updated, fixed, changed dictionaries into your Kobo root into the directory customdict. After this you need to reboot to get the actual dictionaries updated. See this thread for more information. The adaptions and script mentioned in this post are included in the mega update.

WARNINGS If this is the first time you install this patch, you to fix the password for root and disable telnet. This is an important step, here are the steps you have to take (taken from this old post):

Turn on Wifi on the Kobo and find IP address
Go to Settings Connect and after this is done, go to Settings Device Information where you will see something likeIP Address: 192.168.1.NN
(numbers change!)
telnet into your device
telnet 192.168.1.NN
it will ask you the user name, enter root (without the quotes) and no password
(ON THE GLO) change home directory of root
edit /etc/passwd with vi and change the entry for root by changing the 6th field from: / to /root (without the quotes). After this procedure the line should look like
root::0:0:root:/root:/bin/sh
don t forget to save the file
(ON THE GLO) create ssh keys for dropbear
[root@(none) ~]# mkdir /etc/dropbear [root@(none) ~]# cd /etc/dropbear [root@(none) ~]# dropbearkey -t dss -f dropbear_dss_host_key [root@(none) ~]# dropbearkey -t rsa -f dropbear_rsa_host_key
(ON YOUR PERSONAL COMPUTER) check that you can log in with ssh
ssh root@192.168.1.NN
You should get dropped into your device again
(ON THE GLO) log out of the telnet session (the first one you did)
[root@(none) ~]# exit
(ON THE GLO) in your ssh session, change the password of root
[root@(none) ~]# passwd
you will have to enter the new password two times. Remember it well, you will not be easily able to recover it without opening your device.
(ON THE GLO) disable telnet login
edit the file /etc/inetd.conf.local on the GLO (using vi) and remove the telnet line (the line starting with 23).
restart your device

The combined KoboRoot.tgz is provided without warranty. If you need to reset your device, don t blame me!

10 September 2017

Sylvain Beucler: dot-zed archive file format

TL,DR: I reverse-engineered the .zed encrypted archive format.
Following a clean-room design, I'm providing a description that can be implemented by a third-party.
Interested?

(reference version at: https://www.beuc.net/zed/) .zed archive file format Introduction Archives with the .zed extension are conceptually similar to an encrypted .zip file. In addition to a specific format, .zed files support multiple users: files are encrypted using the archive master key, which itself is encrypted for each user and/or authentication method (password, RSA key through certificate or PKCS#11 token). Metadata such as filenames is partially encrypted. .zed archives are used as stand-alone or attached to e-mails with the help of a MS Outlook plugin. A variant, which is not covered here, can encrypt/decrypt MS Windows folders on the fly like ecryptfs. In the spirit of academic and independent research this document provides a description of the file format and encryption algorithms for this encrypted file archive. See the conventions section for conventions and acronyms used in this document. Structure overview The .zed file format is composed of several layers.

The main container is using the (MS-CFB), which is notably used by MS Office 97-2003 .doc files. It contains several streams:
- Metadata stream: in OLE Property Set format (MS-OLEPS), contains 2 blobs in a specific Type-Length-Value (TLV) format:
  - _ctlfile: global archive properties and access list
    It is obfuscated by means of static-key AES encryption.
    The properties include archive initial filename and a global IV.
    A global encryption key is itself encrypted in each user entry.
  - _catalog: file list
    Contains each file metadata indexed with a 15-bytes identifier.
    Directories are supported.
    Full filename is encrypted using AES.
    File extension is (redundantly) stored in clear, and so are file metadata such as modification time.
- Each file in the archive compressed with zlib and encrypted with the standard AES algorithm, in a separate stream.
  Several encryption schemes and key sizes are supported.
  The file stream is split in chunks of 512 bytes, individually encrypted.
- Optional streams, contain additional metadata as well as pictures to display in the application background ("watermarks"). They are not discussed here.

Or as a diagram:

+----------------------------------------------------------------------------------------------------+
  .zed archive (MS-CBF)                                                                               
                                                                                                      
   stream #1                         stream #2                       stream #3...                     
  +------------------------------+  +---------------------------+  +---------------------------+      
    metadata (MS-OLEPS)               encryption (AES)               encryption (AES)                 
                                      512-bytes chunks               512-bytes chunks                 
    +--------------------------+                                                                      
      obfuscation (static key)        +-----------------------+      +-----------------------+        
      +----------------------+       -  compression (zlib)     -    -  compression (zlib)     -       
       _ctlfile (TLV)                                                                            ...  
      +----------------------+          +---------------+              +---------------+               
    +--------------------------+          file contents                  file contents                
                                                                                                      
    +--------------------------+     -  +---------------+      -    -  +---------------+      -       
      _catalog (TLV)                                                                                  
    +--------------------------+      +-----------------------+      +-----------------------+        
  +------------------------------+  +---------------------------+  +---------------------------+      
+----------------------------------------------------------------------------------------------------+

Encryption schemes Several AES key sizes are supported, such as 128 and 256 bits. The Cipher Block Chaining (CBC) block cipher mode of operation is used to decrypt multiple AES 16-byte blocks, which means an initialisation vector (IV) is stored in clear along with the ciphertext. All filenames and file contents are encrypted using the same encryption mode, key and IV (e.g. if you remove and re-add a file in the archive, the resulting stream will be identical). No cleartext padding is used during encryption; instead, several end-of-stream handlers are available, so the ciphertext has exactly the size of the cleartext (e.g. the size of the compressed file). The following variants were identified in the 'encryption_mode' field. STREAM This is the end-of-stream handler for:

obfuscated metadata encrypted with static AES key
filenames and files in archives with 'encryption_mode' set to "AES-CBC-STREAM"
any AES ciphertext of size < 16 bytes, regardless of encryption mode

This end-of-stream handler is apparently specific to the .zed format, and applied when the cleartext's does not end on a 16-byte boundary ; in this case special processing is performed on the last partial 16-byte block. The encryption and decryption phases are identical: let's assume the last partial block of cleartext (for encryption) or ciphertext (for decryption) was appended after all the complete 16-byte blocks of ciphertext:

the second-to-last block of the ciphertext is encrypted in AES-ECB mode (i.e. block cipher encryption only, without XORing with the IV)
then XOR-ed with the last partial block (hence truncated to the length of the partial block)

In either case, if the full ciphertext is less then one AES block (< 16 bytes), then the IV is used instead of the second-to-last block. CTS CTS or CipherText Stealing is the end-of-stream handler for:

filenames and files in archives with 'encryption_mode' set to "AES-CBC-CTS".
- exception: if the size of the ciphertext is < 16 bytes, then "STREAM" is used instead.

It matches the CBC-CS3 variant as described in Recommendation for Block Cipher Modes of Operation: Three Variants of Ciphertext Stealing for CBC Mode. Empty cleartext Since empty filenames or metadata are invalid, and since all files are compressed (resulting in a minimum 8-byte zlib cleartext), no empty cleartext was encrypted in the archive. metadata stream It is named 05356861616161716149656b7a6565636e576a33317a7868304e63 (hexadecimal), i.e. the character with code 5 followed by '5haaaaqaIekzeecnWj31zxh0Nc' (ASCII). The format used is OLE Property Set (MS-OLEPS). It introduces 2 property names "_ctlfile" (index 3) and "_catalog" (index 4), and 2 instances of said properties each containing an application-specific VT_BLOB (type 0x0041). _ctlfile: obfuscated global properties and access list This subpart is stored under index 3 ("_ctlfile") of the MS-OLEPS metadata. It consists of:

static delimiter 0765921A2A0774534752073361719300 (hexadecimal) followed by 0100 (hexadecimal) (18 bytes total)
16-byte IV
ciphertext
1 uint32be representing the length of all the above
static delimiter 0765921A2A0774534752073361719300 (hexadecimal) followed by "ZoneCentral (R)" (ASCII) and a NUL byte (32 bytes total)

The ciphertext is encrypted with AES-CBC "STREAM" mode using 128-bit static key 37F13CF81C780AF26B6A52654F794AEF (hexadecimal) and the prepended IV so as to obfuscate the access list. The ciphertext is continuous and not split in chunks (unlike files), even when it is larger than 512 bytes. The decrypted text contain properties in a TLV format as described in _ctlfile TLV:

global archive properties as a 'fileprops' structure,
extra archive properties as a 'archive_extraprops' structure
users access list as a series of 'passworduser' and 'rsauser entries.

Archives may include "mandatory" users that cannot be removed. They are typically used to add an enterprise wide recovery RSA key to all archives. Extreme care must be taken to protect these key, as it can decrypt all past archives generated from within that company. _catalog: file list This subpart is stored under index 4 ("_catalog") of the MS-OLEPS metadata. It contains a series of 'fileprops' TLV structures, one for each file or directory. The file hierarchy can be reconstructed by checking the 'parent_id' field of each file entry. If 'parent_id' is 0 then the file is located at the top-level of the hierarchy, otherwise it's located under the directory with the matching 'file_id'. TLV format This format is a series of fields :

4 bytes for Type (specified as a 4-bytes hexadecimal below)
4 bytes for value Length (uint32be)
Value

Value semantics depend on its Type. It may contain an uint32be integer, a UTF-16LE string, a character sequence, or an inner TLV structure. Unless otherwise noted, TLV structures appear once. Some fields are optional and may not be present at all (e.g. 'archive_createdwith'). Some fields are unique within a structure (e.g. 'files_iv'), other may be repeated within a structure to form a list (e.g. 'fileprops' and 'passworduser'). The following top-level types that have been identified, and detailed in the next sections:

80110600: fileprops, used for the file list as well as for the global archive properties
001b0600: archive_extraprops
80140600: accesslist

Some additional unidentified types may be present. _ctlfile TLV

80110600: fileprops (TLV structure): global archive properties
- 00230400: archive_pathname (UTF-16LE string): initial archive filename (past versions also leaked the full pathname of the initial archive)
- 80270200: encryption_mode (utf32be): 103 for "AES-CBC-STREAM", 104 for "AES-CBC-CTS"
- 80260200: encryption_strength (utf32be): AES key size, in bytes (e.g. 32 means AES with a 256-bit key)
- 80280500: files_iv (sequence of bytes): global IV for all filenames and file contents
001b0600: archive_extraprops (TLV structure): additionnal archive properties (optional)
- 00c40500: archive_creationtime (FILETIME): date and time when archive was initially created (optional)
- 00c00400: archive_createdwith (UTF-16LE string): uuid-like structure describing the application that initialized the archive (optional)
  00000188-1000-3CA8-8868-36F59DEFD14D is Zed! Free 1.0.188.
80140600: accesslist (TLV structure): describe the users, their key encryption and their permissions
- 80610600: passworduser (TLV structure): user identified by password (0 or more)
- 80620600: rsauser (TLV structure): user identified by RSA key (via file or PKCS#11 token) (0 or more)
- Fields common to passworduser and rsauser:
  - 80710400: login (UTF-16LE string): user name
  - 80720300: login_md5 (sequence of bytes): used by the application to search for a user name
  - 807e0100: priv1 (uchar): user privileges; present and set to 1 when user is admin (optional)
  - 00830200: priv2 (uint32be): user privileges; present and set to 2 when user is admin, present and set to 5 when user is a marked as mandatory, e.g. for recovery keys (optional)
  - 80740500: files_key_ciphertext (sequence of bytes): the archive encryption key, itself encrypted
  - 00840500: user_creationtime (FILETIME): date and time when the user was added to the archive
- passworduser-specific fields:
  - 80760500: pbe_salt (sequence of bytes): salt for PBE
  - 80770200: pbe_iter (uint32be): number of iterations for PBE
  - 80780200: pkcs12_hashfunc (uint32be): hash function used for PBE and PBA key derivation
  - 80790500: pba_checksum (sequence of bytes): password derived with PBA to check for password validity
  - 807a0500: pba_salt (sequence of bytes): salt for PBA
  - 807b0200: pba_iter (uint32be): number of iterations for PBA
- rsauser-specific fields:
  - 807d0500: certificate (sequence of bytes): user X509 certificate in DER format

_catalog TLV

80110600: fileprops (TLV structure): describe the archive files (0 or more)
- 80300500: file_id (sequence of bytes): a 16-byte unique identifier
- 80310400: filename_halfanon (UTF-16LE string): half-anonymized filename, e.g. File1.txt (leaking filename extension)
- 00380500: filename_ciphertext (sequence of bytes): encrypted filename; may have a trailing NUL byte once decrypted
- 80330500: file_size (uint64le): decompressed file size in bytes
- 80340500: file_creationtime (FILETIME): file creation date and time
- 80350500: file_lastwritetime (FILETIME): file last modification date and time
- 80360500: file_lastaccesstime (FILETIME): file last access date and time
- 00370500: parent_directory_id (sequence of bytes): file_id of the parent directory, 0 is top-level
- 80320100: is_dir (uint32be): 1 if entry is directory (optional)

Decrypting the archive AES key rsauser The user accessing the archive will be authenticated by comparing his/her X509 certificate with the one stored in the 'certificate' field using DER format. The 'files_key_ciphertext' field is then decrypted using the PKCS#1 v1.5 encryption mechanism, with the private key that matches the user certificate. passworduser An intermediary user key, a user IV and an integrity checksum will be derived from the user password, using the deprecated PKCS#12 method as described at rfc7292 appendix B. Note: this is not PKCS#5 (nor PBKDF1/PBKDF2), this is an incompatible method from PKCS#12 that notably does not use HMAC. The 'pkcs12_hashfunc' field defines the underlying hash function. The following values have been identified:

21: SHA-1
22: SHA-256

PBA - Password-based authentication The user accessing the archive will be authenticated by deriving an 8-byte sequence from his/her password. The parameters for the derivation function are:

ID: 3
'pba_salt': the salt, typically an 8-byte random sequence
'pba_iter': the iteration count, typically 200000

The derivation is checked against 'pba_checksum'. PBE - Password-based encryption Once the user is identified, 2 new values are derived from the password with different parameters to produce the IV and the key decryption key, with the same hash function:

'pbe_salt': the salt, typically an 8-bytes random sequence
'pbe_iter': the iteration count, typically 100000

The parameters specific to user key are:

ID: 1
size: 32

The user key needs to be truncated to a length of 'encryption_strength', as specified in bytes in the archive properties. The parameters specific to user IV are:

ID: 2
size: 16

Once the key decryption key and the IV are derived, 'files_key_ciphertext' is decrypted using AES CBC, with PKCS#7 padding. Identifying file streams The name of the MS-CFB stream is derived by shuffling the bytes from the 'file_id' field and then encoding the result as hexadecimal. The reordering is:

Initial  offset: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Shuffled offset: 3 2 1 0 5 4 7 6 8 9 10 11 12 13 14 15

The 16th byte is usually a NUL byte, hence the stream identifier is a 30-character-long string. Decrypting files The compressed stream is split in chunks of 512 bytes, each of them encrypted separately using AES CBS and the global archive encryption scheme. Decryption uses the global AES key (retrieved using the user credentials), and the global IV (retrieved from the deobfuscated archive metadata). The IV for each chunk is computed by:

expressing the current chunk number as little endian on 16 bytes
XORing it with the global IV
encrypting with the global AES key in ECB mode (without IV).

Each chunk is an independent stream and the decryption process involves end-of-stream handling even if this is not the end of the actual file. This is particularly important for the CTS handler. Note: this is not to be confused with CTR block cipher mode of operation with operates differently and requires a nonce. Decompressing files Compressed streams are zlib stream with default compression options and can be decompressed following the zlib format. Test cases Excluded for brevity, cf. https://www.beuc.net/zed/#test-cases. Conventions and references

uint32le: unsigned int on 4 bytes, little endian.
uint32be: unsigned int on 4 bytes, big endian.
uint64le: unsigned int on 8 bytes, little endian.
uchar: unsigned char on 1 byte.
(hexadecimal): represents 1 byte with 2 hexadecimal numbers.
e.g. 4142 (hexadecimal) means decimal values 65 and then 66.
(ascii): represents 1 byte with the matching 1-byte ASCII value.
e.g. 'AB\0' means decimal values 65 then 66 and then 0.
UTF-16LE: text encoded using 2-bytes Unicode, little-endian, without Byte Order Mark (BOM), as commonly used under MS Windows.
IV: initialization vector for AES encryption and decryption.
CBC: Cipher Block Chaining, see Recommendation for Block Cipher Modes of Operation and Wikipedia.
CTS: CipherText Stealing, one type of end-of-stream handling, see Recommendation for Block Cipher Modes of Operation: Three Variants of Ciphertext Stealing for CBC Mode and Wikipedia.
PBA, PBE: password-based authentication, password-based encryption; see PKCS #11 Cryptographic Token Interface Current Mechanisms Specification.
MS-CFB: Compound File Binary Format, documented at MSDN, libolecf, Wikipedia.
Libraries that can manipulate this format include olefile, libolecf and libforensics.
MS-OLEPS: OLE Property Set, documented at MSDN and libfole.
The 'oleps.py' demo from libforensics displays detailed information on MS-OLEPS streams; olefile and libolecf have basic support.
TLV : Type-Length-Value encoding ; 4 bytes for type, 4 bytes for length (uint32be), and a variable-sized value; see Type-length-value at Wikipedia for the general principle.
FILETIME: timestamp as 1/10000000s since January 1, 1601 (UTC), see FILETIME structure at MSDN.

Feedback Feel free to send comments at beuc@beuc.net. If you have .zed files that you think are not covered by this document, please send them as well (replace sensitive files with other ones). The author's GPG key can be found at 8FF1CB6E8D89059F. Copyright (C) 2017 Sylvain Beucler Copying and distribution of this file, with or without modification, are permitted in any medium without royalty provided the copyright notice and this notice are preserved. This file is offered as-is, without any warranty.

Sylvain Beucler: dot-zed archive file format

TL,DR: I reverse-engineered the .zed encrypted archive format.
Following a clean-room design, I'm providing a description that can be implemented by a third-party.
Interested?

The main container is using the (MS-CFB), which is notably used by MS Office 97-2003 .doc files. It contains several streams:
- Metadata stream: in OLE Property Set format (MS-OLEPS), contains 2 blobs in a specific Type-Length-Value (TLV) format:
  - _ctlfile: global archive properties and access list
    It is obfuscated by means of static-key AES encryption.
    The properties include archive initial filename and a global IV.
    A global encryption key is itself encrypted in each user entry.
  - _catalog: file list
    Contains each file metadata indexed with a 15-bytes identifier.
    Directories are supported.
    Full filename is encrypted using AES.
    File extension is (redundantly) stored in clear, and so are file metadata such as modification time.
- Each file in the archive compressed with zlib and encrypted with the standard AES algorithm, in a separate stream.
  Several encryption schemes and key sizes are supported.
  The file stream is split in chunks of 512 bytes, individually encrypted.
- Optional streams, contain additional metadata as well as pictures to display in the application background ("watermarks"). They are not discussed here.

Or as a diagram:

+----------------------------------------------------------------------------------------------------+
  .zed archive (MS-CBF)                                                                               
                                                                                                      
   stream #1                         stream #2                       stream #3...                     
  +------------------------------+  +---------------------------+  +---------------------------+      
    metadata (MS-OLEPS)               encryption (AES)               encryption (AES)                 
                                      512-bytes chunks               512-bytes chunks                 
    +--------------------------+                                                                      
      obfuscation (static key)        +-----------------------+      +-----------------------+        
      +----------------------+       -  compression (zlib)     -    -  compression (zlib)     -       
       _ctlfile (TLV)                                                                            ...  
      +----------------------+          +---------------+              +---------------+               
    +--------------------------+          file contents                  file contents                
                                                                                                      
    +--------------------------+     -  +---------------+      -    -  +---------------+      -       
      _catalog (TLV)                                                                                  
    +--------------------------+      +-----------------------+      +-----------------------+        
  +------------------------------+  +---------------------------+  +---------------------------+      
+----------------------------------------------------------------------------------------------------+

obfuscated metadata encrypted with static AES key
filenames and files in archives with 'encryption_mode' set to "AES-CBC-STREAM"
any AES ciphertext of size < 16 bytes, regardless of encryption mode

the second-to-last block of the ciphertext is encrypted in AES-ECB mode (i.e. block cipher encryption only, without XORing with the IV)
then XOR-ed with the last partial block (hence truncated to the length of the partial block)

filenames and files in archives with 'encryption_mode' set to "AES-CBC-CTS".
- exception: if the size of the ciphertext is < 16 bytes, then "STREAM" is used instead.

static delimiter 0765921A2A0774534752073361719300 (hexadecimal) followed by 0100 (hexadecimal) (18 bytes total)
16-byte IV
ciphertext
1 uint32be representing the length of all the above
static delimiter 0765921A2A0774534752073361719300 (hexadecimal) followed by "ZoneCentral (R)" (ASCII) and a NUL byte (32 bytes total)

global archive properties as a 'fileprops' structure,
extra archive properties as a 'archive_extraprops' structure
users access list as a series of 'passworduser' and 'rsauser entries.

4 bytes for Type (specified as a 4-bytes hexadecimal below)
4 bytes for value Length (uint32be)
Value

80110600: fileprops, used for the file list as well as for the global archive properties
001b0600: archive_extraprops
80140600: accesslist

Some additional unidentified types may be present. _ctlfile TLV

80110600: fileprops (TLV structure): global archive properties
- 00230400: archive_pathname (UTF-16LE string): initial archive filename (past versions also leaked the full pathname of the initial archive)
- 80270200: encryption_mode (utf32be): 103 for "AES-CBC-STREAM", 104 for "AES-CBC-CTS"
- 80260200: encryption_strength (utf32be): AES key size, in bytes (e.g. 32 means AES with a 256-bit key)
- 80280500: files_iv (sequence of bytes): global IV for all filenames and file contents
001b0600: archive_extraprops (TLV structure): additionnal archive properties (optional)
- 00c40500: archive_creationtime (FILETIME): date and time when archive was initially created (optional)
- 00c00400: archive_createdwith (UTF-16LE string): uuid-like structure describing the application that initialized the archive (optional)
  00000188-1000-3CA8-8868-36F59DEFD14D is Zed! Free 1.0.188.
80140600: accesslist (TLV structure): describe the users, their key encryption and their permissions
- 80610600: passworduser (TLV structure): user identified by password (0 or more)
- 80620600: rsauser (TLV structure): user identified by RSA key (via file or PKCS#11 token) (0 or more)
- Fields common to passworduser and rsauser:
  - 80710400: login (UTF-16LE string): user name
  - 80720300: login_md5 (sequence of bytes): used by the application to search for a user name
  - 807e0100: priv1 (uchar): user privileges; present and set to 1 when user is admin (optional)
  - 00830200: priv2 (uint32be): user privileges; present and set to 2 when user is admin, present and set to 5 when user is a marked as mandatory, e.g. for recovery keys (optional)
  - 80740500: files_key_ciphertext (sequence of bytes): the archive encryption key, itself encrypted
  - 00840500: user_creationtime (FILETIME): date and time when the user was added to the archive
- passworduser-specific fields:
  - 80760500: pbe_salt (sequence of bytes): salt for PBE
  - 80770200: pbe_iter (uint32be): number of iterations for PBE
  - 80780200: pkcs12_hashfunc (uint32be): hash function used for PBE and PBA key derivation
  - 80790500: pba_checksum (sequence of bytes): password derived with PBA to check for password validity
  - 807a0500: pba_salt (sequence of bytes): salt for PBA
  - 807b0200: pba_iter (uint32be): number of iterations for PBA
- rsauser-specific fields:
  - 807d0500: certificate (sequence of bytes): user X509 certificate in DER format

_catalog TLV

80110600: fileprops (TLV structure): describe the archive files (0 or more)
- 80300500: file_id (sequence of bytes): a 16-byte unique identifier
- 80310400: filename_halfanon (UTF-16LE string): half-anonymized filename, e.g. File1.txt (leaking filename extension)
- 00380500: filename_ciphertext (sequence of bytes): encrypted filename; may have a trailing NUL byte once decrypted
- 80330500: file_size (uint64le): decompressed file size in bytes
- 80340500: file_creationtime (FILETIME): file creation date and time
- 80350500: file_lastwritetime (FILETIME): file last modification date and time
- 80360500: file_lastaccesstime (FILETIME): file last access date and time
- 00370500: parent_directory_id (sequence of bytes): file_id of the parent directory, 0 is top-level
- 80320100: is_dir (uint32be): 1 if entry is directory (optional)

21: SHA-1
22: SHA-256

PBA - Password-based authentication The user accessing the archive will be authenticated by deriving an 8-byte sequence from his/her password. The parameters for the derivation function are:

ID: 3
'pba_salt': the salt, typically an 8-byte random sequence
'pba_iter': the iteration count, typically 200000

'pbe_salt': the salt, typically an 8-bytes random sequence
'pbe_iter': the iteration count, typically 100000

The parameters specific to user key are:

ID: 1
size: 32

The user key needs to be truncated to a length of 'encryption_strength', as specified in bytes in the archive properties. The parameters specific to user IV are:

ID: 2
size: 16

Initial  offset: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Shuffled offset: 3 2 1 0 5 4 7 6 8 9 10 11 12 13 14 15

expressing the current chunk number as little endian on 16 bytes
XORing it with the global IV
encrypting with the global AES key in ECB mode (without IV).

uint32le: unsigned int on 4 bytes, little endian.
uint32be: unsigned int on 4 bytes, big endian.
uint64le: unsigned int on 8 bytes, little endian.
uchar: unsigned char on 1 byte.
(hexadecimal): represents 1 byte with 2 hexadecimal numbers.
e.g. 4142 (hexadecimal) means decimal values 65 and then 66.
(ascii): represents 1 byte with the matching 1-byte ASCII value.
e.g. 'AB\0' means decimal values 65 then 66 and then 0.
UTF-16LE: text encoded using 2-bytes Unicode, little-endian, without Byte Order Mark (BOM), as commonly used under MS Windows.
IV: initialization vector for AES encryption and decryption.
CBC: Cipher Block Chaining, see Recommendation for Block Cipher Modes of Operation and Wikipedia.
CTS: CipherText Stealing, one type of end-of-stream handling, see Recommendation for Block Cipher Modes of Operation: Three Variants of Ciphertext Stealing for CBC Mode and Wikipedia.
PBA, PBE: password-based authentication, password-based encryption; see PKCS #11 Cryptographic Token Interface Current Mechanisms Specification.
MS-CFB: Compound File Binary Format, documented at MSDN, libolecf, Wikipedia.
Libraries that can manipulate this format include olefile, libolecf and libforensics.
MS-OLEPS: OLE Property Set, documented at MSDN and libfole.
The 'oleps.py' demo from libforensics displays detailed information on MS-OLEPS streams; olefile and libolecf have basic support.
TLV : Type-Length-Value encoding ; 4 bytes for type, 4 bytes for length (uint32be), and a variable-sized value; see Type-length-value at Wikipedia for the general principle.
FILETIME: timestamp as 1/10000000s since January 1, 1601 (UTC), see FILETIME structure at MSDN.

18 June 2017

Simon Josefsson: OpenPGP smartcard under GNOME on Debian 9.0 Stretch

I installed Debian 9.0 Stretch on my Lenovo X201 laptop today. Installation went smooth, as usual. GnuPG/SSH with an OpenPGP smartcard I use a YubiKey NEO does not work out of the box with GNOME though. I wrote about how to fix OpenPGP smartcards under GNOME with Debian 8.0 Jessie earlier, and I thought I d do a similar blog post for Debian 9.0 Stretch . The situation is slightly different than before (e.g., GnuPG works better but SSH doesn t) so there is some progress. May I hope that Debian 10.0 Buster gets this right? Pointers to which package in Debian should have a bug report tracking this issue is welcome (or a pointer to an existing bug report). After first login, I attempt to use gpg --card-status to check if GnuPG can talk to the smartcard.

jas@latte:~$ gpg --card-status
gpg: error getting version from 'scdaemon': No SmartCard daemon
gpg: OpenPGP card not available: No SmartCard daemon
jas@latte:~$

This fails because scdaemon is not installed. Isn t a smartcard common enough so that this should be installed by default on a GNOME Desktop Debian installation? Anyway, install it as follows.

root@latte:~# apt-get install scdaemon

Then try again.

jas@latte:~$ gpg --card-status
gpg: selecting openpgp failed: No such device
gpg: OpenPGP card not available: No such device
jas@latte:~$

I believe scdaemon here attempts to use its internal CCID implementation, and I do not know why it does not work. At this point I often recall that want pcscd installed since I work with smartcards in general.

root@latte:~# apt-get install pcscd

Now gpg --card-status works!

jas@latte:~$ gpg --card-status
Reader ...........: Yubico Yubikey NEO CCID 00 00
Application ID ...: D2760001240102000006017403230000
Version ..........: 2.0
Manufacturer .....: Yubico
Serial number ....: 01740323
Name of cardholder: Simon Josefsson
Language prefs ...: sv
Sex ..............: male
URL of public key : https://josefsson.org/54265e8c.txt
Login data .......: jas
Signature PIN ....: not forced
Key attributes ...: rsa2048 rsa2048 rsa2048
Max. PIN lengths .: 127 127 127
PIN retry counter : 3 3 3
Signature counter : 8358
Signature key ....: 9941 5CE1 905D 0E55 A9F8  8026 860B 7FBB 32F8 119D
      created ....: 2014-06-22 19:19:04
Encryption key....: DC9F 9B7D 8831 692A A852  D95B 9535 162A 78EC D86B
      created ....: 2014-06-22 19:19:20
Authentication key: 2E08 856F 4B22 2148 A40A  3E45 AF66 08D7 36BA 8F9B
      created ....: 2014-06-22 19:19:41
General key info..: sub  rsa2048/860B7FBB32F8119D 2014-06-22 Simon Josefsson 
sec#  rsa3744/0664A76954265E8C  created: 2014-06-22  expires: 2017-09-04
ssb>  rsa2048/860B7FBB32F8119D  created: 2014-06-22  expires: 2017-09-04
                                card-no: 0006 01740323
ssb>  rsa2048/9535162A78ECD86B  created: 2014-06-22  expires: 2017-09-04
                                card-no: 0006 01740323
ssb>  rsa2048/AF6608D736BA8F9B  created: 2014-06-22  expires: 2017-09-04
                                card-no: 0006 01740323
jas@latte:~$

Using the key will not work though.

jas@latte:~$ echo foo gpg -a --sign
gpg: no default secret key: No secret key
gpg: signing failed: No secret key
jas@latte:~$

This is because the public key and the secret key stub are not available.

jas@latte:~$ gpg --list-keys
jas@latte:~$ gpg --list-secret-keys
jas@latte:~$

You need to import the key for this to work. I have some vague memory that gpg --card-status was supposed to do this, but I may be wrong.

jas@latte:~$ gpg --recv-keys 9AA9BDB11BB1B99A21285A330664A76954265E8C
gpg: failed to start the dirmngr '/usr/bin/dirmngr': No such file or directory
gpg: connecting dirmngr at '/run/user/1000/gnupg/S.dirmngr' failed: No such file or directory
gpg: keyserver receive failed: No dirmngr
jas@latte:~$

Surprisingly, dirmngr is also not shipped by default so it has to be installed manually.

root@latte:~# apt-get install dirmngr

Below I proceed to trust the clouds to find my key.

jas@latte:~$ gpg --recv-keys 9AA9BDB11BB1B99A21285A330664A76954265E8C
gpg: key 0664A76954265E8C: public key "Simon Josefsson " imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg:               imported: 1
jas@latte:~$

Now the public key and the secret key stub are available locally.

jas@latte:~$ gpg --list-keys
/home/jas/.gnupg/pubring.kbx
----------------------------
pub   rsa3744 2014-06-22 [SC] [expires: 2017-09-04]
      9AA9BDB11BB1B99A21285A330664A76954265E8C
uid           [ unknown] Simon Josefsson 
uid           [ unknown] Simon Josefsson 
sub   rsa2048 2014-06-22 [S] [expires: 2017-09-04]
sub   rsa2048 2014-06-22 [E] [expires: 2017-09-04]
sub   rsa2048 2014-06-22 [A] [expires: 2017-09-04]
jas@latte:~$ gpg --list-secret-keys
/home/jas/.gnupg/pubring.kbx
----------------------------
sec#  rsa3744 2014-06-22 [SC] [expires: 2017-09-04]
      9AA9BDB11BB1B99A21285A330664A76954265E8C
uid           [ unknown] Simon Josefsson 
uid           [ unknown] Simon Josefsson 
ssb>  rsa2048 2014-06-22 [S] [expires: 2017-09-04]
ssb>  rsa2048 2014-06-22 [E] [expires: 2017-09-04]
ssb>  rsa2048 2014-06-22 [A] [expires: 2017-09-04]
jas@latte:~$

I am now able to sign data with the smartcard, yay!

jas@latte:~$ echo foo gpg -a --sign
-----BEGIN PGP MESSAGE-----
owGbwMvMwMHYxl2/2+iH4FzG01xJDJFu3+XT8vO5OhmNWRgYORhkxRRZZjrGPJwQ
yxe68keDGkwxKxNIJQMXpwBMRJGd/a98NMPJQt6jaoyO9yUVlmS7s7qm+Kjwr53G
uq9wQ+z+/kOdk9w4Q39+SMvc+mEV72kuH9WaW9bVqj80jN77hUbfTn5mffu2/aVL
h/IneTfaOQaukHij/P8A0//Phg/maWbONUjjySrl+a3tP8ll6/oeCd8g/aeTlH79
i0naanjW4bjv9wnvGuN+LPHLmhUc2zvZdyK3xttN/roHvsdX3f53yTAxeInvXZmd
x7W0/hVPX33Y4nT877T/ak4L057IBSavaPVcf4yhglVI8XuGgaTP666Wuslbliy4
5W5eLasbd33Xd/W0hTINznuz0kJ4r1bLHZW9fvjLduMPq5rS2co9tvW8nX9rhZ/D
zycu/QA=
=I8rt
-----END PGP MESSAGE-----
jas@latte:~$

Encrypting to myself will not work smoothly though.

jas@latte:~$ echo foo gpg -a --encrypt -r simon@josefsson.org
gpg: 9535162A78ECD86B: There is no assurance this key belongs to the named user
sub  rsa2048/9535162A78ECD86B 2014-06-22 Simon Josefsson 
 Primary key fingerprint: 9AA9 BDB1 1BB1 B99A 2128  5A33 0664 A769 5426 5E8C
      Subkey fingerprint: DC9F 9B7D 8831 692A A852  D95B 9535 162A 78EC D86B
It is NOT certain that the key belongs to the person named
in the user ID.  If you *really* know what you are doing,
you may answer the next question with yes.
Use this key anyway? (y/N) 
gpg: signal Interrupt caught ... exiting
jas@latte:~$

The reason is that the newly imported key has unknown trust settings. I update the trust settings on my key to fix this, and encrypting now works without a prompt.

jas@latte:~$ gpg --edit-key 9AA9BDB11BB1B99A21285A330664A76954265E8C
gpg (GnuPG) 2.1.18; Copyright (C) 2017 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Secret key is available.
pub  rsa3744/0664A76954265E8C
     created: 2014-06-22  expires: 2017-09-04  usage: SC  
     trust: unknown       validity: unknown
ssb  rsa2048/860B7FBB32F8119D
     created: 2014-06-22  expires: 2017-09-04  usage: S   
     card-no: 0006 01740323
ssb  rsa2048/9535162A78ECD86B
     created: 2014-06-22  expires: 2017-09-04  usage: E   
     card-no: 0006 01740323
ssb  rsa2048/AF6608D736BA8F9B
     created: 2014-06-22  expires: 2017-09-04  usage: A   
     card-no: 0006 01740323
[ unknown] (1). Simon Josefsson 
[ unknown] (2)  Simon Josefsson 
gpg> trust
pub  rsa3744/0664A76954265E8C
     created: 2014-06-22  expires: 2017-09-04  usage: SC  
     trust: unknown       validity: unknown
ssb  rsa2048/860B7FBB32F8119D
     created: 2014-06-22  expires: 2017-09-04  usage: S   
     card-no: 0006 01740323
ssb  rsa2048/9535162A78ECD86B
     created: 2014-06-22  expires: 2017-09-04  usage: E   
     card-no: 0006 01740323
ssb  rsa2048/AF6608D736BA8F9B
     created: 2014-06-22  expires: 2017-09-04  usage: A   
     card-no: 0006 01740323
[ unknown] (1). Simon Josefsson 
[ unknown] (2)  Simon Josefsson 
Please decide how far you trust this user to correctly verify other users' keys
(by looking at passports, checking fingerprints from different sources, etc.)
  1 = I don't know or won't say
  2 = I do NOT trust
  3 = I trust marginally
  4 = I trust fully
  5 = I trust ultimately
  m = back to the main menu
Your decision? 5
Do you really want to set this key to ultimate trust? (y/N) y
pub  rsa3744/0664A76954265E8C
     created: 2014-06-22  expires: 2017-09-04  usage: SC  
     trust: ultimate      validity: unknown
ssb  rsa2048/860B7FBB32F8119D
     created: 2014-06-22  expires: 2017-09-04  usage: S   
     card-no: 0006 01740323
ssb  rsa2048/9535162A78ECD86B
     created: 2014-06-22  expires: 2017-09-04  usage: E   
     card-no: 0006 01740323
ssb  rsa2048/AF6608D736BA8F9B
     created: 2014-06-22  expires: 2017-09-04  usage: A   
     card-no: 0006 01740323
[ unknown] (1). Simon Josefsson 
[ unknown] (2)  Simon Josefsson 
Please note that the shown key validity is not necessarily correct
unless you restart the program.
gpg> quit
jas@latte:~$ echo foo gpg -a --encrypt -r simon@josefsson.org
-----BEGIN PGP MESSAGE-----
hQEMA5U1Fip47NhrAQgArTvAykj/YRhWVuXb6nzeEigtlvKFSmGHmbNkJgF5+r1/
/hWENR72wsb1L0ROaLIjM3iIwNmyBURMiG+xV8ZE03VNbJdORW+S0fO6Ck4FaIj8
iL2/CXyp1obq1xCeYjdPf2nrz/P2Evu69s1K2/0i9y2KOK+0+u9fEGdAge8Gup6y
PWFDFkNj2YiVa383BqJ+kV51tfquw+T4y5MfVWBoHlhm46GgwjIxXiI+uBa655IM
EgwrONcZTbAWSV4/ShhR9ug9AzGIJgpu9x8k2i+yKcBsgAh/+d8v7joUaPRZlGIr
kim217hpA3/VLIFxTTkkm/BO1KWBlblxvVaL3RZDDNI5AVp0SASswqBqT3W5ew+K
nKdQ6UTMhEFe8xddsLjkI9+AzHfiuDCDxnxNgI1haI6obp9eeouGXUKG
=s6kt
-----END PGP MESSAGE-----
jas@latte:~$

So everything is fine, isn t it? Alas, not quite.

jas@latte:~$ ssh-add -L
The agent has no identities.
jas@latte:~$

Tracking this down, I now realize that GNOME s keyring is used for SSH but GnuPG s gpg-agent is used for GnuPG. GnuPG uses the environment variable GPG_AGENT_INFO to connect to an agent, and SSH uses the SSH_AUTH_SOCK environment variable to find its agent. The filenames used below leak the knowledge that gpg-agent is used for GnuPG but GNOME keyring is used for SSH.

jas@latte:~$ echo $GPG_AGENT_INFO 
/run/user/1000/gnupg/S.gpg-agent:0:1
jas@latte:~$ echo $SSH_AUTH_SOCK 
/run/user/1000/keyring/ssh
jas@latte:~$

Here the same recipe as in my previous blog post works. This time GNOME keyring only has to be disabled for SSH. Disabling GNOME keyring is not sufficient, you also need gpg-agent to start with enable-ssh-support. The simplest way to achieve that is to add a line in ~/.gnupg/gpg-agent.conf as follows. When you login, the script /etc/X11/Xsession.d/90gpg-agent will set the environment variables GPG_AGENT_INFO and SSH_AUTH_SOCK. The latter variable is only set if enable-ssh-support is mentioned in the gpg-agent configuration.

jas@latte:~$ mkdir ~/.config/autostart
jas@latte:~$ cp /etc/xdg/autostart/gnome-keyring-ssh.desktop ~/.config/autostart/
jas@latte:~$ echo 'Hidden=true' >> ~/.config/autostart/gnome-keyring-ssh.desktop 
jas@latte:~$ echo enable-ssh-support >> ~/.gnupg/gpg-agent.conf 
jas@latte:~$

Log out from GNOME and log in again. Now you should see ssh-add -L working.

jas@latte:~$ ssh-add -L
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFP+UOTZJ+OXydpmbKmdGOVoJJz8se7lMs139T+TNLryk3EEWF+GqbB4VgzxzrGjwAMSjeQkAMb7Sbn+VpbJf1JDPFBHoYJQmg6CX4kFRaGZT6DHbYjgia59WkdkEYTtB7KPkbFWleo/RZT2u3f8eTedrP7dhSX0azN0lDuu/wBrwedzSV+AiPr10rQaCTp1V8sKbhz5ryOXHQW0Gcps6JraRzMW+ooKFX3lPq0pZa7qL9F6sE4sDFvtOdbRJoZS1b88aZrENGx8KSrcMzARq9UBn1plsEG4/3BRv/BgHHaF+d97by52R0VVyIXpLlkdp1Uk4D9cQptgaH4UAyI1vr cardno:000601740323
jas@latte:~$

Topics for further discussion or research include 1) whether scdaemon, dirmngr and/or pcscd should be pre-installed on Debian desktop systems; 2) whether gpg --card-status should attempt to import the public key and secret key stub automatically; 3) why GNOME keyring is used by default for SSH rather than gpg-agent; 4) whether GNOME keyring should support smartcards, or if it is better to always use gpg-agent for GnuPG/SSH, 5) if something could/should be done to automatically infer the trust setting for a secret key. Enjoy!

2 June 2017

Evgeni Golov: Breaking glass, OnePlus service and Android backups

While visiting our Raleigh office, I managed to crack the glass on the screen of my OnePlus 3. Luckily it was a clean crack from the left upper corner, to the right lower one. The crack was not really interfering with neither touch nor display, so I had not much pressure in fixing it. eBay lists new LCD sets for 110-130 , and those still require manual work of getting the LCD assembly out of the case, replacing it, etc. There are also glass-only sets for ~20 , but these require the complete removal of the glued glass part from the screen, and reattaching it, nothing you want to do at home. But there is also still the vendor, who can fix it, right? Internet suggested they would do it for about 100 , which seemed fair. As people have been asking about the support experience, here is a quick write up what happened:

Opened the RMA request online on Sunday, providing a brief description of the issue and some photos
Monday morning answer from the support team, confirming this is way out of warranty, but I can get the device fixed for about 93
After confirming that the extra cost is expected, I had an UPS sticker to ship the device to CTDI in Poland
UPS even tried a pick-up on Tuesday, but I was not properly prepared, so I dropped the device later at a local UPS point
It arrived in Poland on Wednesday
On Thursday the device was inspected, pictures made etc
Friday morning I had a quote in my inbox, asking me to pay 105 - the service partner decided to replace the front camera too, which was not part of the original 93 approximation.
Paid the money with my credit card and started waiting.
The actual repair happened on Monday.
Quality controlled on Tuesday.
Shipped to me on Wednesday.
Arrived at my door on Thursday.

All in all 9 working days, which is not great, but good enough IMHO. And the repair is good, and it was not (too) expensive. So I am a happy user of an OnePlus 3 again. Well, almost. Before sending the device for repairs, had to take a backup and wipe it. I would not send it with my, even encrypted, data on it. And backups and Android is something special. Android will backup certain data to Google, if you allow it to. Apps can forbid that. Sadly this also blocks non-cloud backups with adb backup. So to properly backup your system, you either need root or you create a full backup of the system in the recovery and restore that. I did the backup using TWRP, transferred it to my laptop, wiped the device, sent it in, got it back, copied the backup to the phone, restored it and... Was locked out of the device, it would not take my password anymore. Well, it seems that happens, just delete some files and it will be fine. It's 2017, are backups of mobile devices really supposed to be that hard?!

31 May 2017

Lars Wirzenius: Using a Yubikey 4 for ensafening one's encryption

Introduction I've written before about using a U2F key with PAM. This post continues the theme and explains how to use a smartcard with GnuPG for storing OpenPGP private keys. Specifically, a Yubikey 4 card, because that's what I have, but any good GnuPG compatible card should work. The Yubikey is both a GnuPG compatible smart card, and a U2F card. The Yubikey 4 can handle keys up to 4096 bits. Older Yubikeys can only handle keys up to 2095 bits. The reason to do this is to make it harder for an attacker to steal your encryption keys. I will assume you don't already have an OpenPGP key, or are willing to generate a new one. I will also assume you run Debian stretch; some of the desktop environment setup details may differ between Debian versions or between Linux distributions. You will need:

A Yubikey 4 (or other GnuPG compatible smartcard).
Two USB memory sticks to store master copies of the key you create.
Either a lot of patience, or a ChaosKey or something else to generate a lot of entropy for the kernel. Entropy is used by GnuPG to create encryption keys.

Terminology Some terminology:

OpenPGP is a standard of encryption keys.
GnuPG (also known as GPG or gpg) is an encryption program that supports OpenPGP.
Encryption key or key pair is a secret and a public key for encryption. Key and key pair are often used as synonyms.
The secret keys is yours, and only you will have it or be able to you use. The public key is public, anyone can have a copy. The two are linked and you can't create a secret key linked to a public key. (If you can, you will become a famous cryptographer. Like Isaac Newton famous among apple growers.)
Master key is the important thing to keep track of. In the encryption world, the master key is what identifies you.
A subkey is a secondary key or key pair for encryption, derived from the master key. Subkeys can be created and revoked almost at will, and there can be any number of them. Subkeys are always associated with a single master key.
Subkeys can be dedicated for encryption, signing, and authentication. You can have one of each.
Encryption is the process of taking a file and making it illegible to everyone except the owner of a secret key. When the public key is used to encrypt, only the secret key can decrypt.
Signing is using a secret key to make a separate file that others can verify using the corresponding public key. This means the signer can "prove" they have the file. For example, you might sign emails to prove they came from you.
Authentication is the process of proving you're you. Typically, for example, a server will know your public key, and will give a large random number encrypted with your public key. Only you have the corresponding secret key, so only you can send the random number back to the server, and thereby the server knows you're you.

Outline The process outline is:

Create a new, signing-only master key with GnuPG.
Create three "subkeys", one each for encryption, signing, and authentication. These subkeys are what everyone else uses.
Export copies of the master key pair and the subkey pairs and put them in a safe place.
Put the subkeys on the Yubikey.
GnuPG will automatically use the keys from the card. You have to have the card plugged into a USB port for things to work. If someone steals your laptop, they won't get the private subkeys. Even if they steal your Yubikey, they won't get them (the smartcard is physically designed to prevent that), and can't even use them (because there's PIN codes or passphrases and getting them wrong several times locks up the smartcard).
Use gpg-agent as your SSH agent, and the authentication-only subkey on the Yubikey is used as your ssh key.

Configure GnuPG The process in more detail:

Configure GnuPG with regards to checksum and encryption algorithms. You can use the defaults, but depending on the version of GnuPG you have, they may be weaker than is recommended. These values are from Riseup.net OpenPGP guide, see link at the end. Add the following lines to ~/.gnupg/gpg.conf:
```
  personal-digest-preferences SHA512
  cert-digest-algo SHA512
  default-preference-list SHA512 SHA384 SHA256 SHA224 AES256 AES192 AES CAST5 ZLIB BZIP2 ZIP Uncompressed
  personal-cipher-preferences TWOFISH CAMELLIA256 AES 3DES
  keyserver pool.sks-keyservers.net
```

Create new keys

Create new sign-only master key.

$ gpg --full-generate-key gpg (GnuPG) 2.1.18; Copyright (C) 2017 Free Software Foundation, Inc. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Please select what kind of key you want: (1) RSA and RSA (default) (2) DSA and Elgamal (3) DSA (sign only) (4) RSA (sign only) Your selection? 4 RSA keys may be between 1024 and 4096 bits long. What keysize do you want? (2048) 4096 Requested keysize is 4096 bits Please specify how long the key should be valid. 0 = key does not expire = key expires in n days w = key expires in n weeks m = key expires in n months y = key expires in n years Key is valid for? (0) 1y Key expires at Tue 29 May 2018 06:43:54 PM EEST Is this correct? (y/N) y GnuPG needs to construct a user ID to identify your key. Real name: Lars Wirzenius Email address: liw@liw.fi Comment: test key You selected this USER-ID: "Lars Wirzenius (test key) <liw@liw.fi>>" Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o We need to generate a lot of random bytes. It is a good idea to perform some other action (type on the keyboard, move the mouse, utilize the disks) during the prime generation; this gives the random number generator a better chance to gain enough entropy. gpg: key 25FB738D6EE435F7 marked as ultimately trusted gpg: directory '/home/liw/.gnupg/openpgp-revocs.d' created gpg: revocation certificate stored as '/home/liw/.gnupg/openpgp-revocs.d/A734C10BF2DF39D19DC0F6C025FB738D6EE435F7.rev' public and secret key created and signed. Note that this key cannot be used for encryption. You may want to use the command "--edit-key" to generate a subkey for this purpose. pub rsa4096 2017-05-29 [SC] [expires: 2018-05-29] A734C10BF2DF39D19DC0F6C025FB738D6EE435F7 A734C10BF2DF39D19DC0F6C025FB738D6EE435F7 uid Lars Wirzenius (test key) <liw@liw.fi>

Note that I set a 1-year expiration for they key. The expiration can be extended at any time (if you have the master secret key), but unless you do, the key won't accidentally live longer than the chosen time.

Review the key:

$ gpg --list-secret-keys /home/liw/.gnupg/pubring.kbx ---------------------------- sec rsa4096 2017-05-29 [SC] [expires: 2018-05-29] A734C10BF2DF39D19DC0F6C025FB738D6EE435F7 uid [ultimate] Lars Wirzenius (test key) <liw@liw.fi>

You now have the signing-only master key. You should now create three subkeys (keyid is the key identifier shown in the key listing, A734C10BF2DF39D19DC0F6C025FB738D6EE435F7 above). Use the --expert option to be able to add an authentication-only subkey.

$ gpg --edit-key --expert A734C10BF2DF39D19DC0F6C025FB738D6EE435F7z gpg (GnuPG) 2.1.18; Copyright (C) 2017 Free Software Foundation, Inc. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Secret key is available. sec rsa4096/25FB738D6EE435F7 created: 2017-05-29 expires: 2018-05-29 usage: SC trust: ultimate validity: ultimate [ultimate] (1). Lars Wirzenius (test key) <liw@liw.fi> gpg> addkey Please select what kind of key you want: (3) DSA (sign only) (4) RSA (sign only) (5) Elgamal (encrypt only) (6) RSA (encrypt only) (7) DSA (set your own capabilities) (8) RSA (set your own capabilities) (10) ECC (sign only) (11) ECC (set your own capabilities) (12) ECC (encrypt only) (13) Existing key Your selection? 4 RSA keys may be between 1024 and 4096 bits long. What keysize do you want? (2048) 4096 Requested keysize is 4096 bits Please specify how long the key should be valid. 0 = key does not expire = key expires in n days w = key expires in n weeks m = key expires in n months y = key expires in n years Key is valid for? (0) 1y Key expires at Tue 29 May 2018 06:44:52 PM EEST Is this correct? (y/N) y Really create? (y/N) y We need to generate a lot of random bytes. It is a good idea to perform some other action (type on the keyboard, move the mouse, utilize the disks) during the prime generation; this gives the random number generator a better chance to gain enough entropy. sec rsa4096/25FB738D6EE435F7 created: 2017-05-29 expires: 2018-05-29 usage: SC trust: ultimate validity: ultimate ssb rsa4096/05F88308DFB71774 created: 2017-05-29 expires: 2018-05-29 usage: S [ultimate] (1). Lars Wirzenius (test key) <liw@liw.fi> gpg> addkey Please select what kind of key you want: (3) DSA (sign only) (4) RSA (sign only (5) Elgamal (encrypt only) (6) RSA (encrypt only) (7) DSA (set your own capabilities) (8) RSA (set your own capabilities) (10) ECC (sign only) (11) ECC (set your own capabilities) (12) ECC (encrypt only) (13) Existing key Your selection? 6 RSA keys may be between 1024 and 4096 bits long. What keysize do you want? (2048) 4096 Requested keysize is 4096 bits Please specify how long the key should be valid 0 = key does not expire = key expires in n days w = key expires in n weeks m = key expires in n months y = key expires in n years Key is valid for? (0) 1y Key expires at Tue 29 May 2018 06:45:22 PM EEST Is this correct? (y/N) y Really create? (y/N) y We need to generate a lot of random bytes. It is a good idea to perform some other action (type on the keyboard, move the mouse, utilize the disks) during the prime generation; this gives the random number generator a better chance to gain enough entropy. sec rsa4096/25FB738D6EE435F7 created: 2017-05-29 expires: 2018-05-29 usage: SC trust: ultimate validity: ultimate ssb rsa4096/05F88308DFB71774 created: 2017-05-29 expires: 2018-05-29 usage: S ssb rsa4096/2929E8A96CBA57C7 created: 2017-05-29 expires: 2018-05-29 usage: E [ultimate] (1). Lars Wirzenius (test key) <liw@liw.fi> gpg> addkey Please select what kind of key you want: (3) DSA (sign only) (4) RSA (sign only) (5) Elgamal (encrypt only) (6) RSA (encrypt only) (7) DSA (set your own capabilities) (8) RSA (set your own capabilities) (10) ECC (sign only) (11) ECC (set your own capabilities) (12) ECC (encrypt only) (13) Existing key Your selection? 8 Possible actions for a RSA key: Sign Encrypt Authenticate Current allowed actions: Sign Encrypt (S) Toggle the sign capability (E) Toggle the encrypt capability (A) Toggle the authenticate capability (Q) Finished Your selection? a Possible actions for a RSA key: Sign Encrypt Authenticate Current allowed actions: Sign Encrypt Authenticate (S) Toggle the sign capability (E) Toggle the encrypt capability (A) Toggle the authenticate capability (Q) Finished Your selection? s Possible actions for a RSA key: Sign Encrypt Authenticate Current allowed actions: Encrypt Authenticate (S) Toggle the sign capability (E) Toggle the encrypt capability (A) Toggle the authenticate capability (Q) Finished Your selection? e Possible actions for a RSA key: Sign Encrypt Authenticate Current allowed actions: Authenticate (S) Toggle the sign capability (E) Toggle the encrypt capability (A) Toggle the authenticate capability (Q) Finished Your selection? q RSA keys may be btween 1024 and 4096 bits long. What keysize do you want? (2048) 4096 Requested keysize is 4096 bits Please specify how long the key should be valid. 0 = key does not expire = key expires in n days w = key expires in n weeks m = key expires in n months y = key expires in n years Key is valid for? (0) 1y Key expires at Tue 29 May 2018 06:45:56 PM EEST Is this correct? (y/N) y Really create? (y/N) y We need to generate a lot of random bytes. It is a good idea to perform some other action (type on the keyboard, move the mouse, utilize the disks) during the prime generation; this gives the random number generator a better chance to gain enough entropy. sec rsa4096/25FB738D6EE435F7 created: 2017-05-29 expires: 2018-05-29 usage: SC trust: ultimate validity: ultimate ssb rsa4096/05F88308DFB71774 created: 2017-05-29 expires: 2018-05-29 usage: S ssb rsa4096/2929E8A96CBA57C7 created: 2017-05-29 expires: 2018-05-29 usage: E ssb rsa4096/4477EB0AEF1C440A created: 2017-05-29 expires: 2018-05-29 usage: A [ultimate] (1). Lars Wirzenius (test key) <liw@liw.fi> gpg> save
Export secret keys to files, make a backup

You now have a master key and three subkeys. They are hidden in the ~/.gnupg directory. It is time to "export" the secret keys out from there.

$ gpg --export-secret-key --armor keyid > master.key $ gpg --export-secret-subkeys --armor keyid > subkeys.key

You should keep these files safe. You don't want to lose them, and you don't want anyone else to get access to them. I recommend you format two USB memory sticks, format them using full-disk encryption, and copy the exported files to both of them. Then keep them somewhere safe. There's ways of making this part more sophisticated, but that's for another time.

The next step involves some hoop-jumping. What we want is to have the master secret key NOT on you machine, so we tell GnuPG to remove it. We exported it above, so we won't lose it. However, deleting the master secret key also removes the secret subkeys. But we can import those without importing the master secret key.

$ gpg --delete-secret-key keyid $ gpg --import subkeys.key

Now verify that you have the secret subkeys, but not the master key. There should be one line starting with sec# (note the hash mark, which indicates the key isn't available), and three lines starting with ssb (no hash mark).

$ gpg -K /home/liw/.gnupg/pubring.kbx ---------------------------- sec# rsa4096 2017-05-29 [SC] [expires: 2018-05-29] A734C10BF2DF39D19DC0F6C025FB738D6EE435F7 uid [ultimate] Lars Wirzenius (test key) <liw@liw.fi> ssb rsa4096 2017-05-29 [S] [expires: 2018-05-29] ssb rsa4096 2017-05-29 [E] [expires: 2018-05-29] ssb rsa4096 2017-05-29 [A] [expires: 2018-05-29]
Install subkeys on a Yubikey

Now insert the Yubikey in a USB slot. We can start transferring the secret subkeys to the Yubikey. If you want, you can set your name and other information, and change PIN codes. There's several types of PIN codes: normal use, unblocking a locked card, and a third PIN code for admin operations. Changing the PIN codes is a good idea, otherwise everyone will just try the default of 123456 (admin 12345678). However, I'm skipping that in the interest of brevity.

$ gpg -card-edit ...

Actually move the subkeys to the card. Note that this does a move, not a copy, and the subkeys will be removed from your ~/.gnupg (check with gpg -K).

$ gpg --edit-key liw gpg (GnuPG) 2.1.18; Copyright (C) 2017 Free Software Foundation, Inc. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Secret key is available. pub rsa4096/25FB738D6EE435F7 created: 2017-05-29 expires: 2018-05-29 usage: SC trust: ultimate validity: ultimate ssb rsa4096/05F88308DFB71774 created: 2017-05-29 expires: 2018-05-29 usage: S ssb rsa4096/2929E8A96CBA57C7 created: 2017-05-29 expires: 2018-05-29 usage: E ssb rsa4096/4477EB0AEF1C440A created: 2017-05-29 expires: 2018-05-29 usage: A [ultimate] (1). Lars Wirzenius (test key) <liw@liw.fi> gpg> key 1 pub rsa4096/25FB738D6EE435F7 created: 2017-05-29 expires: 2018-05-29 usage: SC trust: ultimate validity: ultimate ssb* rsa4096/05F88308DFB71774 created: 2017-05-29 expires: 2018-05-29 usage: S ssb rsa4096/2929E8A96CBA57C7 created: 2017-05-29 expires: 2018-05-29 usage: E ssb rsa4096/4477EB0AEF1C440A created: 2017-05-29 expires: 2018-05-29 usage: A [ultimate] (1). Lars Wirzenius (test key) <liw@liw.fi> gpg> keytocard Please select where to store the key: (1) Signature key (3) Authentication key Your selection? 1 pub rsa4096/25FB738D6EE435F7 created: 2017-05-29 expires: 2018-05-29 usage: SC trust: ultimate validity: ultimate ssb* rsa4096/05F88308DFB71774 created: 2017-05-29 expires: 2018-05-29 usage: S ssb rsa4096/2929E8A96CBA57C7 created: 2017-05-29 expires: 2018-05-29 usage: E ssb rsa4096/4477EB0AEF1C440A created: 2017-05-29 expires: 2018-05-29 usage: A [ultimate] (1). Lars Wirzenius (test key) <liw@liw.fi> gpg> key 1 pub rsa4096/25FB738D6EE435F7 created: 2017-05-29 expires: 2018-05-29 usage: SC trust: ultimate validity: ultimate ssb rsa4096/05F88308DFB71774 created: 2017-05-29 expires: 2018-05-29 usage: S ssb rsa4096/2929E8A96CBA57C7 created: 2017-05-29 expires: 2018-05-29 usage: E ssb rsa4096/4477EB0AEF1C440A created: 2017-05-29 expires: 2018-05-29 usage: A [ultimate] (1). Lars Wirzenius (test key) <liw@liw.fi> gpg> key 2 pub rsa4096/25FB738D6EE435F7 created: 2017-05-29 expires: 2018-05-29 usage: SC trust: ultimate validity: ultimate ssb rsa4096/05F88308DFB71774 created: 2017-05-29 expires: 2018-05-29 usage: S ssb* rsa4096/2929E8A96CBA57C7 created: 2017-05-29 expires: 2018-05-29 usage: E ssb rsa4096/4477EB0AEF1C440A created: 2017-05-29 expires: 2018-05-29 usage: A [ultimate] (1). Lars Wirzenius (test key) <liw@liw.fi> gpg> keytocard Please select where to store the key: (2) Encryption key Your selection? 2 pub rsa4096/25FB738D6EE435F7 created: 2017-05-29 expires: 2018-05-29 usage: SC trust: ultimate validity: ultimate ssb rsa4096/05F88308DFB71774 created: 2017-05-29 expires: 2018-05-29 usage: S ssb* rsa4096/2929E8A96CBA57C7 created: 2017-05-29 expires: 2018-05-29 usage: E ssb rsa4096/4477EB0AEF1C440A created: 2017-05-29 expires: 2018-05-29 usage: A [ultimate] (1). Lars Wirzenius (test key) <liw@liw.fi> gpg> key 2 pub rsa4096/25FB738D6EE435F7 created: 2017-05-29 expires: 2018-05-29 usage: SC trust: ultimate validity: ultimate ssb rsa4096/05F88308DFB71774 created: 2017-05-29 expires: 2018-05-29 usage: S ssb rsa4096/2929E8A96CBA57C7 created: 2017-05-29 expires: 2018-05-29 usage: E ssb rsa4096/4477EB0AEF1C440A created: 2017-05-29 expires: 2018-05-29 usage: A [ultimate] (1). Lars Wirzenius (test key) <liw@liw.fi> gpg> key 3 pub rsa4096/25FB738D6EE435F7 created: 2017-05-29 expires: 2018-05-29 usage: SC trust: ultimate validity: ultimate ssb rsa4096/05F88308DFB71774 created: 2017-05-29 expires: 2018-05-29 usage: S ssb rsa4096/2929E8A96CBA57C7 created: 2017-05-29 expires: 2018-05-29 usage: E ssb* rsa4096/4477EB0AEF1C440A created: 2017-05-29 expires: 2018-05-29 usage: A [ultimate] (1). Lars Wirzenius (test key) <liw@liw.fi> gpg> keytocard Please select where to store the key: (3) Authentication key Your selection? 3 pub rsa4096/25FB738D6EE435F7 created: 2017-05-29 expires: 2018-05-29 usage: SC trust: ultimate validity: ultimate ssb rsa4096/05F88308DFB71774 created: 2017-05-29 expires: 2018-05-29 usage: S ssb rsa4096/2929E8A96CBA57C7 created: 2017-05-29 expires: 2018-05-29 usage: E ssb* rsa4096/4477EB0AEF1C440A created: 2017-05-29 expires: 2018-05-29 usage: A [ultimate] (1). Lars Wirzenius (test key) <liw@liw.fi> gpg> save

If you want to use several Yubikeys, or have a spare one just in case, repeat the previous four steps (starting from importing subkeys back into ~/.gnupg).

You're now done, as far GnuPG use is concerned. Any time you need to sign, encrypt, or decrypt something, GnuPG will look for your subkeys on the Yubikey, and will tell you to insert it in a USB port if it can't find the key.

Use subkey on Yubikey as your SSH key

To actually use the authentication-only subkey on the Yubikey for ssh, you need to configure your system to use gpg-agent as the SSH agent. Add the following line to .gnupg/gpg-agent.conf:
enable-ssh-support

On a Debian stretch system with GNOME, edit /etc/xdg/autostart/gnome-keyring-ssh.desktop to have the following line, to prevent the GNOME ssh agent from starting up:
Hidden=true

Edit /etc/X11/Xsession.options and remove or comment out the line that says use-ssh-agent. This stops a system-started ssh-agent from being started when the desktop start.

Create the file ~/.config/autostart/gpg-agent.desktop with the following content:
[Desktop Entry] Type=Application Name=gpg-agent Comment=gpg-agent Exec=/usr/bin/gpg-agent --daemon OnlyShowIn=GNOME;Unity;MATE; X-GNOME-Autostart-Phase=PreDisplayServer X-GNOME-AutoRestart=false X-GNOME-Autostart-Notify=true X-GNOME-Bugzilla-Bugzilla=GNOME X-GNOME-Bugzilla-Product=gnome-keyring X-GNOME-Bugzilla-Component=general X-GNOME-Bugzilla-Version=3.20.0

To test, log out, and back in again, run the following in a terminal:

$ ssh-add -l
The output should contain a line that looks like this:
4096 SHA256:PDCzyQPpd9tiWsELM8LwaLBsMDMm42J8/eEfezNgnVc cardno:000604626953 (RSA)

You need to export the authentication-only subkey in the SSH key format. You need this for adding to .ssh/authorized_keys, if nothing else.

$ gpg --export-ssh-key keyid > ssh.pub

Happy hacking.

See also See also the following links. I've used them to learn enough to write the above.

Riseup.net's OpenPGP best practices

bootc's GnuPG / ssh agent setup

anarcat's Yubikey NEO howto

Edited to fix:

Output of gpg -K after removing secret master key.

17 March 2017

Shirish Agarwal: Science Day at GMRT, Khodad 2017

The above picture is the blend of the two communities from foss community and mozilla India. And unless you were there you wouldn t know who is from which community which is what FOSS is all about. But as always I m getting a bit ahead of myself. Akshat, who works at NCRA as a programmer, the standing guy on the left shared with me in January this year that this year too, we should have two stalls, foss community and mozilla India stalls next to each other. While we had the banners, we were missing stickers and flyers. Funds were and are always an issue and this year too, it would have been emptier if we didn t get some money saved from last year minidebconf 2016 that we had in Mumbai. Our major expenses included printing stickers, stationery and flyers which came to around INR 5000/- and couple of LCD TV monitors which came for around INR 2k/- as rent. All the labour was voluntary in nature, but both me and Akshat easily spending upto 100 hours before the event. Next year, we want to raise to around INR 10-15k so we can buy 1 or 2 LCD monitors and we don t have to think for funds for next couple of years. How will we do that I have no idea atm. Me and Akshat did all the printing and stationery runs and hence had not been using my lappy for about 3-4 days. Come to the evening before the event and the laptop would not start. Coincidentally, or not few months or even last at last year s Debconf people had commented on IBM/Lenovo s obsession with proprietary power cords and adaptors. I hadn t given it much thought but when I got no power even after putting it on AC power for 3-4 hours, I looked up on the web and saw that the power cord and power adaptors were all different even in T440 and even that under existing models. In fact I couldn t find mine hence sharing it via pictures below. I knew/suspected that thinkpads would be rare where I was going, it would be rarer still to find the exact power cord and I was unsure whether it was the power cord at fault or adaptor or whatever goes for SMPS in laptop or memory or motherboard/CPU itself. I did look up the documentation at support.lenovo.com and was surprised at the extensive documentation that Lenovo has for remote troubleshooting. I did the usual take out the battery, put it back in, twiddle with the little hole in the bottom of the laptop, trying to switch on without the battery on AC mains, trying to switch on with battery power only but nothing worked. Couple of hours had gone by and with a resigned thought went to bed, convincing myself that anyways it s good I am not taking the lappy as it is extra-dusty there and who needs a dead laptop anyways. Update After the event was over, I did contact Lenovo support and within a week, with one visit from a service engineer, he was able to identify that it was a faulty cable which was at fault and not the the other things which I was afraid of. Another week gone by and lenovo replaced the cable. Going by service standards that I have seen of other companies, Lenovo deserves a gold star here for the prompt service they provided. I probably would end up subscribing to their extended 2-year warranty service when my existing 3 year warranty is about to be over. Next day, woke up early morning, two students from COEP hostel were volunteering and we made our way to NCRA, Pune University Campus. Ironically, though we were under the impression that we would be the late arrivals, it turned out we were the early birds. 5-10 minutes passed by and soon enough we were joined by Aniket and we played catch-up for a while. We hadn t met each other for a while so it was good to catch-up. Then slowly other people starting coming in and around 07:10-07:15 we started for GMRT, Khodad. Now I had been curious as had been hearing for years that the Pune-Nashik NH-50 highway would be concreted and widened to six-lane highways but the experience was below par. Came back and realized the proposal has now been pushed back to 2020. From the mozilla team, only Aniket was with us, the rest of the group was coming straight from Nashik. Interestingly, all the six people who came, came on bikes which depending upon how you look at it was either brave or stupid. Travelling on bikes on Indian highways you either have to be brave or stupid or both, we have more than enough accidents due to quality of road construction, road design, lane-changing drivers and many other issues. This is probably not the place for it hence will use some other blog post to rant about that. We reached around 10:00 hrs. IST and hung around till lunch as Akshat had all the marketing material, monitors etc. The only thing we had were couple of lappies and couple of SBC s, an RPI 3 and a BBB. Our find for the event was Aarti Kashyap who you can see above. She is a third-year student at COEP and one of the rare people who chose to interact with hardware rather than software. From last several years, we had been trying, successfully and unsuccessfully to get more Indian women and girls interested into technology. It is a vicious circle as till a girl/woman doesn t volunteer we are unable to share our knowledge to the extent we can which leads them to not have much interest in FOSS or even technology in general. While there are groups are djangogirls, Pyladies and railgirls and even Outreachy which tries to motivate getting girls into computing but it s a long road ahead. We are short of both funds and ideas as to how to motivate more girls to get into computing and then to get into playing with hardware. I don t know where to start and end for whoever wants to play with hardware. From SBC s, routers to blade servers the sky is the limit. Again this probably isn t the place for it, hence probably we can chew it on more at some other blog post. This year, we had a lowish turnout due to the fact that the 12th board exams 1st paper was on the day we had opened. So instead of 20-25k, we probably had 5-7k fewer people pass through. There were two-three things that we were showing, we were showing Debian on one of the systems, we were showing the output from the SBC s on the other monitor but the glare kept hitting the monitors. While the organizers had done exemplary work over last year. They had taped the carpets on the ground so there was hardly any dust moving around. However, I wished the organizers had taken the pains to have two cloth roofs over our head instead of just one, the other roof head could be say 2 feet up, this would have done two things a. It probably would have cooled the place a bit more as b. We could get diffused sunlight which would have lessened the glare and reflection the LCD s kept throwing back. At times we also got people to come to our side as can be seen in Aarti s photo as can be seen above. If these improvements can be made for next year, this would result in everybody in our Pandal would benefit, not just us and mozilla. This would be benefiting around 10-15 organizations which were within the same temporary structure. Of course, it depends very much on the budget they are able to have and people who are executing, we can just advise. The other thing which had been missing last year and this year is writing about Single Board Computers in Marathi. If we are to promote them as something to replace a computer or something for a younger brother/sister to learn computing upon at a lower cost, we need leaflets written in their language to be more effective. And this needs to be in the language and mannerisms that people in that region understand. India, as probably people might have experienced is a dialect-prone country. Which means every 2-5 kms, the way the language is spoken is different from anywhere else. The Marathi spoken by somebody who has lived in Ravivar Peth for his whole life and a person who has lived in say Kothrud are different. The same goes from any place and this place, Khodad, Narayangaon would have its own dialect, its own mini-codespeak. Just to share, we did have one in English but it would have been a vast improvement if we could do it in the local language. Maybe we can discuss about this and ask for help from people. Outside, Looking in What had been interesting about the whole journey were the new people who were bringing all their passion and creativity to the fore. From the mozilla community, we had Akshay who is supposed to be a wizard on graphics, animation, editing anything to do with the visual medium. He shared some of the work he had done and also shared a bit about how blender works with people who wanted to learn about that. Mayur, whom you see in the picture pointing out something about FOSS and this was the culture that we strove to have. I know and love and hate the browser but haven t been able to fathom the recklessness that Mozilla has been doing the last few years, which has just been having one mis-adventure after another. For instance, mozstumbler was an effort which I thought would go places. From what little I understood, it served/serves as a user-friendly interface to a potential user while still sharing all the data with OSM . They (Mozilla) seems/seemed to have a fatalistic take as it provided initial funding but then never fully committing to the project. Later, at night we had the whole free software and open-source sharings where I tried to emphasize that without free software, the term open-source would not have come into existence. We talked and talked and somewhere around 02:00 I slept, the next day was an extension of the first day itself where we ribbed each other good-naturedly and still shared whatever we could share with each other. I do hope that we continue this tradition for great many years to come and engage with more and more people every passing year.
Filed under: Miscellenous Tagged: #budget, #COEP< #volunteering, #debian, #Events, #Expenses, #mozstumbler, #printing, #SBC's, #Science Day 2017, #thinkpad cable issue, FOSS, mozilla

20 February 2017

Ritesh Raj Sarraf: Setting up appliances - the new way

I own a Fitbit Surge. But Fitibit chose to remain exclusive in terms of interoperability. Which means to make any sense out of the data that the watch gathers, you need to stick with what Fitbit mandates. Fair enough in today's trends. It also is part of their business model to restrict useful aspects of the report to Premium Membership. Again, fair enough in today's business' trends. But a nice human chose to write a bridge; to extract Fitbit data and feed into Google Fit. The project is written in Python, so you can get it to work on most common computer platforms. I never bothered to package this tool for Debian, because I never was sure when I'd throw away the Fitbit. But until that happens, I decided to use the tool to sync my data to Google Fit. Which led me to requirements.txt This project's requirement.txt lists versioned module dependencies, of which many modules in Debian, were either older or newer than what was mentioned in the requirements. To get the tool working, I installed it the pip way. 3 months later, something broke and I needed to revisit the installed modules. At that point, I realized that there's no such thing as: pip upgrade That further led me to dig on why anyone wouldn't add something so simple, because today, in the days of pip, snap, flatpak and dockers, Distributions are predicted to go obsolete and irrelevant. Users should get the SOURCES directly from the developers. But just looking at the date the bug was filed, killed my enthusiasm any further. So, without packaging for Debian, and without installing through pip, I was happy that my init has the ability to create confined and containerized environments, something that I could use to get the job done.
rrs@chutzpah:~$ sudo machinectl login fitbit [sudo] password for rrs: Connected to machine fitbit. Press ^] three times within 1s to exit session. Debian GNU/Linux 9 fitbit pts/0 fitbit login: root Last login: Fri Feb 17 12:44:25 IST 2017 on pts/1 The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. root@fitbit:~# tail -n 25 /var/tmp/lxc/fitbit-google.log synced calories - 1440 data points ------------------------------ 2017-02-19 ------------------------- synced steps - 1440 data points synced distance - 1440 data points synced heart_rate - 38215 data points synced weight - 0 logs synced body_fat - 0 logs synced calories - 1440 data points ------------------------------ 2017-02-20 ------------------------- synced steps - 1270 data points synced distance - 1270 data points synced heart_rate - 32547 data points synced weight - 0 logs synced body_fat - 0 logs synced calories - 1271 data points Synced 7 exercises between : 2017-02-15 -- 2017-02-20 -------------------------------------------------------------------------- Like it ? star the repository : https://github.com/praveendath92/fitbit-googlefit -------------------------------------------------------------------------- root@fitbit:~#

Categories:
Debian-Blog
Computing
Tools
Keywords:
fitbit
pip
python
linux containers
systemd-nspawn
RHUT
Like:

6 February 2017

Daniel Stender: Howto create a Debian 9 preview as Vagrant box with Packer

I ve got some little scripts and a template here to automatically create Vagrant boxes from cutting edge Debian testing daily snapshots (netinstall ISO image) using HashiCorp s Packer. To create Vagrant boxes with these, you first need a running binary of Packer. There is a Debian package available if that s also your working environment, but Packer is going to be introduced into the stable branch with the upcoming Stretch release itself. However, Ubuntu already has it, and some other derivatives, too. And there are prebuild binaries available from the developer s site which run fine out-of-the-box (you just have to put the single binary somewhere into you $PATH, or expand that to find it). The JSON template should run with any Packer which is available for any of the different systems. Vagrant itself isn t needed to build the box with Packer, but Virtualbox is of course needed to pre bake the machine image within a virtual machine. In Debian the base binaries of Virtualbox are in the contrib archive section, so that source might be added to /etc/apt/sources.list, if haven t already. The scripts have been tested to run with 5.1.10, and I haven t seen that any late versions are demanded in particular, but of course heavily outdated versions might not work properly. Packer installs the guest additions ISO file for Virtualbox into the virtual machine (and the shipped provisioning script then builds them inside). For that, the Debian package which ships that (which is in non-free) is recognized if it is installed, and then could be used by Packer. When the ISO isn t available in the places which Packer checks on the host the builder then automatically downloads the corresponding ISO from http://download.virtualbox.org/virtualbox. When the tarball with the scripts is unpacked, just do make create and the process should run through, if Packer and Virtualbox are available. If your environment doesn t have GNU Make nor wget you might want to copy the relevant lines from the Makefile and run it manually. But if it does, just do it like this:
/tmp/debian-testing-vagrantbox$ make create virtualbox-iso output will be in this color. ==> virtualbox-iso: Downloading or copying Guest additions virtualbox-iso: Downloading or copying: file:///usr/share/virtualbox/VBoxGuestAdditions.iso ==> virtualbox-iso: Downloading or copying ISO virtualbox-iso: Downloading or copying: http://cdimage.debian.org/cdimage/daily-builds/daily/arch-latest/amd64/iso-cd/debian-testing-amd64-netinst.iso virtualbox-iso: Download progress: 10% ... virtualbox-iso: Download progress: 96% ==> virtualbox-iso: Starting HTTP server on port 8219 ==> virtualbox-iso: Creating virtual machine... ==> virtualbox-iso: Creating hard drive... ==> virtualbox-iso: Creating forwarded port mapping for communicator (SSH, WinRM, etc) (host port 2885) ==> virtualbox-iso: Starting the virtual machine... ==> virtualbox-iso: Waiting 10s for boot... ==> virtualbox-iso: Typing the boot command... ==> virtualbox-iso: Waiting for SSH to become available...
The Virtualbox window then pops up and the build process continues within the virtual machine for a while. You might want to file a Github issue when there s a problem on your machine, please! (please include the tail of your packer.log) The Packer template (debian-testing-vagrant.json) is described in the documentation of the virtualbox-iso builder. A preseeding script for the Debian Installer (preseed.cfg) is also included which gets injected into the virtual build environment during the build process. The creation progress of the Debian base installation could be easily monitored since the Virtualbox window is fully shown during the Packer run (if you loose your mouse pointer by clicking inside that window, do <Right>+<Control> to escape). For good performance, a fast internet connection is needed since a whole base system must be downloaded if that s available the whole automated process very only takes a couple of minutes to complete on a non-SSD machine. When Packer has finished and a fresh box is created (the size is about 690 MB), it then could be used with Vagrant. Just add the new box with:
/tmp/debian-testing-vagrantbox$ vagrant box add stretch-preview debian-testing-vagrant.box ==> box: Box file was not detected as metadata. Adding it directly... ==> box: Adding box 'stretch-preview' (v0) for provider: box: Unpacking necessary files from: file:///tmp/debian-testing-vagrantbox/debian-testing-vagrant.box ==> box: Successfully added box 'stretch-preview' (v0) for 'virtualbox'!
It then could be initialized within a random working directory with:
/tmp/myproject$ vagrant init stretch-preview A Vagrantfile has been placed in this directory. You are now ready to vagrant up your first virtual environment! Please read the comments in the Vagrantfile as well as documentation on vagrantup.com for more information on using Vagrant.
After that, you could launch the virtual box with:
/tmp/myproject$ vagrant up Bringing machine 'default' up with 'virtualbox' provider... ==> default: Importing base box 'stretch-preview'... ==> default: Matching MAC address for NAT networking... ==> default: Setting the name of the VM: myproject_default_1486321215067_75270 ==> default: Clearing any previously set network interfaces... ==> default: Preparing network interfaces based on configuration... default: Adapter 1: nat ==> default: Forwarding ports... default: 22 (guest) => 2222 (host) (adapter 1) ==> default: Booting VM... ==> default: Waiting for machine to boot. This may take a few minutes... default: SSH address: 127.0.0.1:2222 default: SSH username: vagrant default: SSH auth method: private key default: default: Vagrant insecure key detected. Vagrant will automatically replace default: this with a newly generated keypair for better security. default: default: Inserting generated public key within guest... default: Removing insecure key from the guest if it's present... default: Key inserted! Disconnecting and reconnecting using new SSH key... ==> default: Machine booted and ready! ==> default: Checking for guest additions in VM... ==> default: Mounting shared folders... default: /vagrant => /tmp/myproject /tmp/myproject$
Then you can SSH into it by doing (touch is used here only to point to the shared folder):
/tmp/myproject$ touch hello! /tmp/myproject$ vagrant ssh The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. /usr/bin/xauth: file /home/vagrant/.Xauthority does not exist vagrant@packer-virtualbox-iso-1486319595:~$ $ cat /etc/debian_version 9.0 vagrant@packer-virtualbox-iso-1486319595:~$ ls /vagrant/ hello! Vagrantfile vagrant@packer-virtualbox-iso-1486319595:~$ exit logout Connection to 127.0.0.1 closed. /tmp/myproject$ vagrant halt ==> default: Attempting graceful shutdown of VM... /tmp/myproject$
If you haven t worked with Vagrant before, maybe this is appealing. The user experience is somehwhat different from working with a chroot. Packer makes it very convenient to keep freshly created boxes for it coming. Have fun!

2 February 2017

Shirish Agarwal: The $100 used laptop and getting riled up.

Lenovo-ThinkPad-T500 Source Wikimedia commons
I was reading a thread on phoronix where a student was sharing that it is or can be expensive to get even a used laptop and he shared his predicament and was hammered a bit for it to some going to the extent of questioning his life-choices. While I m not a student it still triggered something in me. I am not dirt poor but neither am I insanely rich. The same questions he has, similar questions I have had. While in his case he is probably in his early to late 20 s, I am pushing 40. Most of the money I make goes in for everyday purchases, veggies, house-rent, electricity, landline, broadband and cell phone bills. What little is left is most of the time kept for a rainy day as there is no Government pension. From what I have heard and read on the web, in the west specifically in the States, if I buy a used laptop, I usually get a 6 months 1 year warranty . Here, while you could get a used laptop for around INR 10k there is no warranty/guarantee, so I never get into that. It s buyer s beware all the time. For people who like/want FOSS or specifically something like Free DOS (like me), I had to wait for almost 6 years to get a model I was happy with, with the specs. I was ok with. Was really lucky enough to get a Thinkpad T440 with 8 GB of RAM for around INR 80k/- with Free DOS. The specs T440 Core I 5 (4300) / Dos (NEW MODEL) 20B7A1SD00 Intel Core i5 4300M (2.5 GHz / 3 MB / 5 GT/s) / Intel QM87 Chipset / Integrated 802.11 n WIFI LAN + Bluetooth 4.0 / 8GB DDR III Memory (2 DIMM SLOT) / 500 GB SATA HDD @ 7200 RPM / 14.0 HDy / FPR / Dos / 2 USB , VGA Port , RJ 45 Port /GB LAN /Track Point with 5 button Glass Touch Pad /Stereo Speakers with Dolby Enhanced Audio / 6 Cell Battery /Approx 2.14KG/ While it is/was actually pretty expensive but then wanted something which can take a beating, deal with all the heat, noise and dust (specifically where I live, right in the middle of the city). The reason I used the word lucky is that now there is no model in the T-series range which has FreeDOS on it. Of course, I hopefully will use it for another 4-5 years at the very least depending on how much it cooperates with me, I have heard that Thinkpads function for a long period of time even in dusty environments so banking on that. What probably pissed me is the condensing note in the comment, how does he know what pressures an another individual might be in. It s almost like saying You are refugee because you made a wrong life choice or something to that effect which again is stupid. I actually feel/felt embarrassed to bring this up as I truly am lucky to be safe, secure, have food on the table, am able to sleep on a bed at night, have a workstation AND a laptop, have somewhat of a sound mind and a body which is able to move around without any hassles. Add to that, incredibly as it may sound, was also able to see another country for a few days In relation to people being persecuted and having to run off to save their own lives or even people living on the streets, I am actually living in luxury. While I can t go through life feeling guilty for all the good things that have happened with me, I do feel disgusted when I see some people put blinding statements like that. One of the biggest reasons that GNU/Linux and Debian in particular gelled with me was that it s incredibly flexible and generous. Nobody tells me which packages I should or shouldn t have. I do right things, good, I do something wrong, an opportunity to learn and hopefully learn from my mistakes. In either case, one of the most forgiving kind of system to learn and hack on. While speaking of mistakes, could somebody look at #849684 . It almost feels like a tennis match going between the maintainers concerned. While I don t have the technical skills to ascertain who s right and who is not, it would be nice if some cooler heads can make sense and see if a way could be found out. Can somebody help ?
Filed under: Miscellenous Tagged: #debian, #Life Choices, #Thinkpad440, laptop

15 November 2016

Keith Packard: AltOS-Lisp

A Tiny Lisp for AltOS I took a bit of a diversion over the last week or so when I wondered how small a lisp interpreter I could write, and whether I could fit that into one of the processors that AltOS runs on. It turns out, you can write a tiny lisp interpreter that fits in about 25kB of ram with a 3kB heap for dynamic data. I decided to target our ChaosKey boards; they're tiny, and I've got a lot of them. That processor offers 28kB of usable flash space (after the 4kB boot loader) and 6kB of ram with the processor running at a steaming 48MHz. I'm not at all sure this is useful, but I always enjoy doing language implementations, and this one presented some 'interesting' challenges:

Limited RAM. I don't have space to do a classic stop/copy collector.

Limited stack. A simple lisp implementation uses the C stack for all recursion in execution and memory collection. I don't have enough ram for that.

Iterative Compacting Allocator I'm betting someone has built one of these before, but I couldn't find one, so I wrote my own. The basic strategy is to walk the heap to find a subset of the active objects which are allocated sequentially in memory with only unused storage between them. These objects are then compacted in-place, and then the heap is walked again to update all references to the moved objects. Then, the process is restarted to find another subset and move them. By looking for these subsets starting at the bottom of the heap, and working upwards towards the top, the whole heap can be compacted into a contiguous chunk at the bottom of memory. Allocation involves moving a pointer along at the top of active memory; when it gets to the top of the heap, collect and see if there's space now. As always, the hardest part was to make sure all active memory was tied down. The second hardest part was to make sure that all active pointers were updated after any allocation, in case a collect moved the underlying object. That was just bookkeeping, but did consume much of the development time. One additional trick was to terminate the recursion during heap walking by flagging active cons cell locations in a global bitmap and then walking that separately, iterating until that bitmap is empty. Nested lambdas form another recursion which should probably get the same approach, but I haven't done that yet. An unexpected "benefit" of the tiny heap is that the collector gets called a lot, so any referencing bugs will have a good chance of being uncovered in even a short program execution. ROM-able Lisp Instead of implementing all of the language in C, I wanted to be able to implement various pieces in Lisp itself. Because of the complex nature of the evaluation process, adding things like 'let' or even 'defun' turn out to be dramatically simpler in Lisp. However, I didn't want to consume bunches of precious RAM to hold these basic functions. What I did was to create two heaps, one in ROM and the other in RAM. References are be tagged as to which heap they're in. 16-bit Values Lisp programs use a pile of references. Using a full 32 bits for each one would mean having a lot less effective storage. So, instead, I use an offset from the base of the heap. The top bit of the offset is used to distinguish between the ROM heap and the RAM heap. I needed a place to store type information, so I settled on using the bottom two bits of the references. This allows for four direct type values. One of these values is used to indicate an indirect type, where the type is stored in the first byte of the object. The direct types are:

Value Type

0 Cons cell

1 14-bit int

2 String

3 Other

With 2 tag bits, the allocator needs to work in 32-bit units as the references couldn't point to individual bytes. Finally, I wanted 0 to be nil, so I add four to the offsets within the heaps. The result is that the ROM and RAM heaps can each cover up to 32k - 4 bytes. Note that ints are not stored in the heap; instead they are immediate values stored in 14 bits, providing a range of -8192 to 8191. One can imagine wanting more range in ints at some point. Heap-based Evaluator A simple lisp implementation uses the fact that eval is re-entrant and do the operation on the C stack:
val eval(val exprs) val vals; while (exprs) vals = append(vals, eval(car(exprs))); exprs = exprs->cdr; return execute (car(vals), cdr(vals));
This makes things really simple and provides for a clean framework for implementing various bits of lisp, including control flow and macros. However, it rapidly consumes all of the available memory for a stack, while also requiring separate bookkeeping for the in-use memory in each frame. I replaced this design with one which keeps the lisp stack on the heap, and then performs eval with a state machine with all state stored in global variables so that the memory manager can reference them directly. Each eval operation is performed in a separate 'stack' context, which holds the entire eval state except for the current value, which lives in a separate global variable and is used to pass values out of one stack frame and into another. When the last stack context is finished, the evaluation terminates and the value is returned to the caller. There are nine states in the state machine, each of which is implemented in a separate function, making the state machine a simple matter of pulling the current state from the top of the stack and invoking the associated function:
while (ao_lisp_stack) if (!(*evals[ao_lisp_stack->state])() ao_lisp_exception) ao_lisp_stack_clear(); return AO_LISP_NIL; return ao_lisp_v;
Because there's no C recursion involved, catching exceptions is a simple matter of one test at this level. Primitives like progn, while, cond and eval all take special magic in the state machine to handle; getting all of that working took several attempts before I found the simple loop shown above. Lexical Scoping The last time I did a lisp interpreter, I implemented dynamic scoping. Atoms were all global and had values associated directly with them. Evaluating a lambda started by saving all of the existing global values for the parameter atoms and then binding the new values. When finished, the previous values would be restored. This is almost correct, but provides surprising results for things like:
> (setq baz 1) > (def foo (lambda (bar) (+ baz bar))) > (def bletch (lambda (baz) (foo baz))) > (bletch 2) 4
The value that foo gets for 'baz' is 2 instead of 1 under dynamic scoping, which most people find surprising. This time, I was determined to use lexical scoping, and it turned out to be surprisingly easy. The first trick was to separate the atoms from their 'value'; each atom can have a different value in different lexical scopes. So, each lexical scope gets a 'frame' object, those contain the value for each atom defined in that scope. There's a global scope which holds all of the globally defined values (like baz, foo and bletch above). Each frame points to its enclosing scope, so you can search upwards to find the right value. The second trick was to realize that the lexical scope of a lambda is the scope in which the lambda itself is evaluated, and that the evaluation of a lambda expression results in a 'function' object, which contains the lambda and its enclosing scope:
> (def foo (lambda (bar bletch) ((lambda (baz) (+ baz bar)) bletch))) > (foo 2 3) 5
In this case, the inner lambda in foo can 'see' the value of bar from the enclosing lambda. More subtly, even if the inner lambda were executed multiple times, it would see the same baz, and could even change it. This can be used to implement all kinds of craziness, including generators:
> (defun make-inc (add) ((lambda (base) (lambda () (progn (setq base (+ base add)) base))) 0) ) > (setq plus2 (make-inc 2)) > (plus2) 2 > (plus2) 4
The current implementation of each frame is a simple array of atom/value pairs, with a reference to the parent frame to form the full scope. There are dramatically faster implementations of this same concept, but the goal here was small and simple. A Tiny Allocator Optimization With eval consuming heap space for stacks, frames and argument lists, the interpreter was spending a lot of time in the collector. As a simple optimization, I added some free lists for stack frames and cons cells. Stack frames are never referenced when they're finished, so they can always go on the free list. Cons cells used to construct argument lists for functions are usually free. Builtin functions have a bit which indicates whether they might hold on to a reference to the argument list. Interpreted lambdas can't get the list while nlambdas, lexprs and macros do. Each lambda execution creates a new frame, and while it would be possible to discover if that frame 'escapes' the lambda, I decided to not attempt to cache free ones yet. Save and Restore To make the lisp interpreter more useful in tiny computers, I added the ability to save and restore the entire heap to flash. This requires leaving enough space in the flash to preserve the heap, further constraining the amount of flash available for the application. Code All of this code is in the 'lisp' branch of my AltOS repository: AltOS The lisp interpreter is independent from the rest of AltOS and could be re-purposed for another embedded operating system. It runs fine on ChaosKey hardware, and also on the STM32F042 Nucleo-32 board There's also a test framework which runs on Linux, and is how I developed almost all of the code. That's in the src/test directory in the above repository, and is called 'ao_lisp_test'. Towers of Hanoi Here's an implementation of the classic recursive Towers of Hanoi game; it shows most of the current features of the language.
; ; Towers of Hanoi ; ; Copyright 2016 Keith Packard <keithp@keithp.com> ; ; This program is free software; you can redistribute it and/or modify ; it under the terms of the GNU General Public License as published by ; the Free Software Foundation, either version 2 of the License, or ; (at your option) any later version. ; ; This program is distributed in the hope that it will be useful, but ; WITHOUT ANY WARRANTY; without even the implied warranty of ; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU ; General Public License for more details. ; ; ANSI control sequences (defun move-to (col row) (patom "\033[" row ";" col "H" nil) ) (defun clear () (patom "\033[2J" nil) ) (defun display-string (x y str) (move-to x y) (patom str) ) ; Here's the pieces to display (setq stack '("*" "**" "***" "****" "*****" "******" "*******")) (setq top (+ (length stack) 3)) ; ; Here's all of the stacks of pieces ; This is generated when the program is run ; (setq stacks nil) ; Display one stack, clearing any ; space above it (defun display-stack (x y clear stack) (cond ((= 0 clear) (cond (stack (progn (display-string x y (car stack)) (display-stack x (1+ y) 0 (cdr stack)) ) ) ) ) (t (progn (display-string x y " ") (display-stack x (1+ y) (1- clear) stack) ) ) ) ) ; This should probably be included in the rom image... (defun length (list) (cond (list (1+ (length (cdr list)))) (0) ) ) ; Position of the top of the stack on the screen ; Shorter stacks start further down the screen (defun stack-pos (y stack) (- y (length stack)) ) ; Display all of the stacks, spaced 20 columns apart (defun display-stacks (x y stacks) (cond (stacks (progn (display-stack x 0 (stack-pos y (car stacks)) (car stacks)) (display-stacks (+ x 20) y (cdr stacks))) ) ) ) ; Display all of the stacks, then move the cursor ; out of the way and flush the output (defun display () (display-stacks 0 top stacks) (move-to 1 21) (flush) ) ; Reset stacks to the starting state, with ; all of the pieces in the first stack and the ; other two empty (defun reset-stacks () (setq stacks (list stack nil nil)) (length stack) ) ; more functions which could usefully ; be in the rom image (defun min (a b) (cond ((< a b) a) (b) ) ) (defun nth (list n) (cond ((= n 0) (car list)) ((nth (cdr list) (1- n))) ) ) ; Replace a stack in the list of stacks ; with a new value (defun replace (list pos member) (cond ((= pos 0) (cons member (cdr list))) ((cons (car list) (replace (cdr list) (1- pos) member))) ) ) ; Move a piece from the top of one stack ; to the top of another (defun move-piece (from to) (let ((from-stack (nth stacks from)) (to-stack (nth stacks to)) (piece (car from-stack))) (setq from-stack (cdr from-stack)) (setq to-stack (cons piece to-stack)) (setq stacks (replace stacks from from-stack)) (setq stacks (replace stacks to to-stack)) (display) (delay 100) ) ) ; The implementation of the game (defun _hanoi (n from to use) (cond ((= 1 n) (progn (move-piece from to) nil) ) (t (progn (_hanoi (1- n) from use to) (_hanoi 1 from to use) (_hanoi (1- n) use to from) ) ) ) ) ; A pretty interface which ; resets the state of the game, ; clears the screen and runs ; the program (defun hanoi () (setq len (reset-stacks)) (clear) (_hanoi len 0 1 2) )

6 November 2016

Russell Coker: Is a Thinkpad Still Like a Rolls-Royce

For a long time the Thinkpad has been widely regarded as the Rolls-Royce of laptops . Since 2003 one could argue that Rolls-Royce is no longer the Rolls-Royce of cars [1]. The way that IBM sold the Think business unit to Lenovo and the way that Lenovo is producing both Thinkpads and cheaper Ideapads is somewhat similar to the way the Rolls-Royce trademark and car company were separately sold to companies that are known for making cheaper cars. Sam Varghese has written about his experience with Thinkpads and how he thinks it s no longer the Rolls-Royce of laptops [2]. Sam makes some reasonable points to support this claim (one of which only applies to touchpad users not people like me who prefer the Trackpoint), but I think that the real issue is whether it s desirable to have a laptop that could be compared to a Rolls-Royce nowadays. Support The Rolls-Royce car company is known for great reliability and support as well as features that other cars lack (mostly luxury features). The Thinkpad marque (both before and after it was sold to Lenovo) was also known for great support. You could take a Thinkpad to any service center anywhere in the world and if the serial number indicated that it was within the warranty period it would be repaired without any need for paperwork. The Thinkpad service centers never had any issue with repairing a Thinkpad that lacked a hard drive just as long as the problem could be demonstrated. It was also possible to purchase an extended support contract at any time which covered all repairs including motherboard replacement. I know that not everyone had as good an experience as I had with Thinkpad support, but I ve been using them since 1998 without problems which is more than I can say for most hardware. Do we really need great reliability from laptops nowadays? When I first got a laptop hardly anyone I knew owned one. Nowadays laptops are common. Having a copy of important documents on a USB stick is often a good substitute for a reliable laptop, when you are in an environment where most people own laptops it s usually not difficult to find someone who will let you use theirs for a while. I think that there is a place for a laptop with RAID-1 and ECC RAM, it s a little known fact that Thinkpads have a long history of supporting the replacement of a CD/DVD drive with a second hard drive (I don t know if this is still supported) but AFAIK they have never supported ECC RAM. My first Thinkpad cost $3,800. In modern money that would be something like $7,000 or more. For that price you really want something that s well supported to protect the valuable asset. Sam complains about his new Thinkpad costing more than $1000 and needing to be replaced after 2.5 years. Mobile phones start at about $600 for the more desirable models (IE anything that runs Pokemon Go) and the new Google Pixel phones range from $1079 to $1,419. Phones aren t really expected to be used for more than 2.5 years. Phones are usually impractical to service in any way so for most of the people who read my blog (who tend to buy the more expensive hardware) they are pretty much a disposable item costing $600+. I previously wrote about a failed Nexus 5 and the financial calculations for self-insuring an expensive phone [3]. I think there s no way that a company can provide extended support/warranty while making a profit and offering a deal that s good value to customers who can afford to self-insure. The same applies for the $499 Lenovo Ideapad 310 and other cheaper Lenovo products. Thinkpads (the higher end of the Lenovo laptop range) are slightly more expensive than the most expensive phones but they also offer more potential for the user to service them. Features My first Thinkpad was quite underpowered when compared to desktop PCs, it had 32M of RAM and could only be expanded to 96M at a time when desktop PCs could be expanded to 128M easily and 256M with some expense. It had a 800*600 display when my desktop display was 1280*1024 (37% of the pixels). Nowadays laptops usually start at about 8G of RAM (with a small minority that have 4G) and laptop displays start at about 1366*768 resolution (51% of the pixels in a FullHD display). That compares well to desktop systems and also is capable of running most things well. My current Thinkpad is a T420 with 8G of RAM and a 1600*900 display (69% of FullHD), it would be nice to have higher resolution but this works well and it was going cheap when I needed a new laptop. Modern Thinkpads don t have some of the significant features that older ones had. The legendary Butterfly Keyboard is long gone, killed by the wide displays that economies of scale and 16:9 movies have forced upon us. It s been a long time since Thinkpads had some of the highest resolution displays and since anyone really cared about it (you only need pixels to be small enough that you can t see them). For me one of the noteworthy features of the Thinkpads has been the great keyboard. Mechanical keys that feel like a desktop keyboard. It seems that most Thinkpads are getting the rubbery keyboard design made popular by Apple. I guess this is due to engineering factors in designing thin laptops and the fact that most users don t care. Matthew Garrett has blogged about the issue of Thinkpad storage configured as RAID mode without any option to disable it [4]. This is an annoyance (which incidentally has been worked around) and there are probably other annoyances like it. Designing hardware and an OS are both complex tasks. The interaction between Windows and the hardware is difficult to get right from both sides and the people who design the hardware often don t think much about Linux support. It has always been this way, the early Thinkpads had no Linux support for special IBM features (like fan control) and support for ISA-PnP was patchy. It is disappointing that Lenovo doesn t put a little extra effort into making sure that Linux works well on their hardware and this might be a reason for considering another brand. Service Life I bought my curent Thinkpad T420 in October 2013 [5] It s more than 3 years old and has no problems even though I bought it refurbished with a reduced warranty. This is probably the longest I ve had a Thinkpad working well, which seems to be a data point against the case that modern Thinkpads aren t as good. I bought a T61 in February 2010 [6], it started working again (after mysteriously not working for a month in late 2013) and apart from the battery lasting 5 minutes and a CPU cooling problem it still works well. If that Thinkpad had cost $3,800 then I would have got it repaired, but as it cost $796 (plus the cost of a RAM upgrade) and a better one was available for $300 it wasn t worth repairing. In the period 1998 to 2010 I bought a 385XD, a 600E, a T21, a T43, and a T61 [6]. During that time I upgraded laptops 4 times in 12 years (I don t have good records of when I bought each one). So my average Thinkpad has lasted 3 years. The first 2 were replaced to get better performance, the 3rd was replaced when an employer assigned me a Thinkpad (and sold it to be when I left), and 4 and 5 were replaced due to hardware problems that could not be fixed economically given the low cost of replacement. Conclusion Thinkpads possibly don t have the benefits over other brands that they used to have. But in terms of providing value for the users it seems that they are much better than they used to be. Until I wrote this post I didn t realise that I ve broken a personal record for owning a laptop. It just keeps working and I hadn t even bothered looking into the issue. For some devices I track how long I ve owned them while thinking can I justify replacing it yet , but the T420 just does everything I want. The battery still lasts 2+ hours which is a new record too, with every other Thinkpad I ve owned the battery life has dropped to well under an hour within a year of purchase. If I replaced this Thinkpad T420 now it will have cost me less than $100 per year (or $140 per year including the new SSD I installed this year), that s about 3 times better than any previous laptop! I wouldn t feel bad about replacing it as I ve definitely got great value for money from it. But I won t replace it as it s doing everything I want. I ve just realised that by every measure (price, reliability, and ability to run all software I want to run) I ve got the best Thinkpad I ve ever had. Maybe it s not like a Rolls-Royce, but I d much rather drive a 2016 Tesla than a 1980 Rolls-Royce anyway.

[1] https://en.wikipedia.org/wiki/Rolls-Royce_Motor_Cars

[2] http://www.itwire.com/open-sauce/75569-thinkpad-no-longer-the-rolls-royce-of-laptops.html

[3] https://etbe.coker.com.au/2016/10/22/another-broken-nexus-5/

[4] http://mjg59.dreamwidth.org/44694.html

[5] https://etbe.coker.com.au/2013/10/23/thinkpad-t420/

[6] https://etbe.coker.com.au/2010/03/16/thinkpad-t61/

[7] https://etbe.coker.com.au/2011/09/08/laptop-to-cloud-lifestyle/

Related posts:

Thinkpad T420 I ve owned a Thinkpad T61 since February 2010 [1]. In...

PC prices drop again! A few weeks ago Dell advertised new laptops for $849AU,...

I Just Bought a new Thinkpad and the Lenovo Web Site Sucks I ve just bought a Thinkpad T61 at auction for $AU796....

2 November 2016

Enrico Zini: schroot connector for ansible

I accidentally made an ansible connector plugin for schroot. I haven't even used ansible yet, so I have no idea what I am doing. You can choose the chroot type by setting something like schroot_type: jessie in your variables. You can install this locally as connection_plugins/schroot.py.
# Based on chroot.py (c) 2013, Maykel Moya <mmoya@speedyrails.com> # Based on chroot.py (c) 2015, Toshio Kuratomi <tkuratomi@ansible.com> # (c) 2016, Enrico Zini <enrico@debian.org> # # This is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # Ansible is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with Ansible. If not, see <http://www.gnu.org/licenses/>. from __future__ import (absolute_import, division, print_function) __metaclass__ = type import distutils.spawn import os import os.path import pipes import subprocess import traceback from ansible import constants as C from ansible.errors import AnsibleError from ansible.plugins.connection import ConnectionBase, BUFSIZE from ansible.module_utils.basic import is_executable from ansible.utils.unicode import to_bytes try: from __main__ import display except ImportError: from ansible.utils.display import Display display = Display() class Connection(ConnectionBase): ''' Local chroot based connections ''' transport = 'schroot' has_pipelining = True # su currently has an undiagnosed issue with calculating the file # checksums (so copy, for instance, doesn't work right) # Have to look into that before re-enabling this become_methods = frozenset(C.BECOME_METHODS).difference(('su',)) def __init__(self, play_context, new_stdin, *args, **kwargs): super(Connection, self).__init__(play_context, new_stdin, *args, **kwargs) self.chroot = self._play_context.remote_addr #if os.geteuid() != 0: #raise AnsibleError("chroot connection requires running as root") # we're running as root on the local system so do some # trivial checks for ensuring 'host' is actually a chroot'able dir #if not os.path.isdir(self.chroot): # raise AnsibleError("%s is not a directory" % self.chroot) #chrootsh = os.path.join(self.chroot, 'bin/sh') ## Want to check for a usable bourne shell inside the chroot. ## is_executable() == True is sufficient. For symlinks it ## gets really complicated really fast. So we punt on finding that ## out. As long as it's a symlink we assume that it will work #if not (is_executable(chrootsh) or (os.path.lexists(chrootsh) and os.path.islink(chrootsh))): # raise AnsibleError("%s does not look like a chrootable dir (/bin/sh missing)" % self.chroot) self.chroot_cmd = distutils.spawn.find_executable('schroot') if not self.chroot_cmd: raise AnsibleError("chroot command not found in PATH") self.chroot_id = "session:" + self.chroot self.chroot_type = "stable" existing = subprocess.check_output([self.chroot_cmd, "--list", "--all"]) self.chroot_exists = False for line in existing.splitlines(): if line == self.chroot_id: self.chroot_exists = True def set_host_overrides(self, host, hostvars=None): super(Connection, self).set_host_overrides(host, hostvars) self.chroot_type = hostvars.get("schroot_type", self.chroot_type) def _connect(self): ''' connect to the chroot; nothing to do here ''' super(Connection, self)._connect() if not self._connected: if not self.chroot_exists: self.chroot_id = subprocess.check_output([self.chroot_cmd, "-b", "-c", self.chroot_type, "-n", self.chroot]).strip() subprocess.check_call([self.chroot_cmd, "-r", "-c", self.chroot_id, "-u", "root", "--", "apt-get", "update"]) subprocess.check_call([self.chroot_cmd, "-r", "-c", self.chroot_id, "-u", "root", "--", "apt-get", "-y", "install", "python"]) display.vvv("THIS IS A LOCAL CHROOT DIR", host=self.chroot) self._connected = True def _buffered_exec_command(self, cmd, stdin=subprocess.PIPE): ''' run a command on the chroot. This is only needed for implementing put_file() get_file() so that we don't have to read the whole file into memory. compared to exec_command() it looses some niceties like being able to return the process's exit code immediately. ''' executable = C.DEFAULT_EXECUTABLE.split()[0] if C.DEFAULT_EXECUTABLE else '/bin/sh' local_cmd = [self.chroot_cmd, "-r", "-c", self.chroot_id, "-u", "root", "--", executable, '-c', cmd] display.vvv("EXEC %s" % (local_cmd), host=self.chroot) local_cmd = [to_bytes(i, errors='strict') for i in local_cmd] p = subprocess.Popen(local_cmd, shell=False, stdin=stdin, stdout=subprocess.PIPE, stderr=subprocess.PIPE) return p def exec_command(self, cmd, in_data=None, sudoable=False): ''' run a command on the chroot ''' super(Connection, self).exec_command(cmd, in_data=in_data, sudoable=sudoable) p = self._buffered_exec_command(cmd) stdout, stderr = p.communicate(in_data) return (p.returncode, stdout, stderr) def _prefix_login_path(self, remote_path): ''' Make sure that we put files into a standard path If a path is relative, then we need to choose where to put it. ssh chooses $HOME but we aren't guaranteed that a home dir will exist in any given chroot. So for now we're choosing "/" instead. This also happens to be the former default. Can revisit using $HOME instead if it's a problem ''' if not remote_path.startswith(os.path.sep): remote_path = os.path.join(os.path.sep, remote_path) return os.path.normpath(remote_path) def put_file(self, in_path, out_path): ''' transfer a file from local to chroot ''' super(Connection, self).put_file(in_path, out_path) display.vvv("PUT %s TO %s" % (in_path, out_path), host=self.chroot) out_path = pipes.quote(self._prefix_login_path(out_path)) try: with open(to_bytes(in_path, errors='strict'), 'rb') as in_file: try: p = self._buffered_exec_command('dd of=%s bs=%s' % (out_path, BUFSIZE), stdin=in_file) except OSError: raise AnsibleError("chroot connection requires dd command in the chroot") try: stdout, stderr = p.communicate() except: traceback.print_exc() raise AnsibleError("failed to transfer file %s to %s" % (in_path, out_path)) if p.returncode != 0: raise AnsibleError("failed to transfer file %s to %s:\n%s\n%s" % (in_path, out_path, stdout, stderr)) except IOError: raise AnsibleError("file or module does not exist at: %s" % in_path) def fetch_file(self, in_path, out_path): ''' fetch a file from chroot to local ''' super(Connection, self).fetch_file(in_path, out_path) display.vvv("FETCH %s TO %s" % (in_path, out_path), host=self.chroot) in_path = pipes.quote(self._prefix_login_path(in_path)) try: p = self._buffered_exec_command('dd if=%s bs=%s' % (in_path, BUFSIZE)) except OSError: raise AnsibleError("chroot connection requires dd command in the chroot") with open(to_bytes(out_path, errors='strict'), 'wb+') as out_file: try: chunk = p.stdout.read(BUFSIZE) while chunk: out_file.write(chunk) chunk = p.stdout.read(BUFSIZE) except: traceback.print_exc() raise AnsibleError("failed to transfer file %s to %s" % (in_path, out_path)) stdout, stderr = p.communicate() if p.returncode != 0: raise AnsibleError("failed to transfer file %s to %s:\n%s\n%s" % (in_path, out_path, stdout, stderr)) def close(self): ''' terminate the connection; nothing to do here ''' super(Connection, self).close() #subprocess.check_command([self.chroot_cmd, "-e", "-c", self.chroot_id]) self._connected = False

23 October 2016

Jaldhar Vyas: What I Did During My Summer Vacation

Thats So Raven If I could sum up the past year in one word, that word would be distraction. There have been so many strange, confusing or simply unforseen things going on I have had trouble focusing like never before. For instance, on the opposite side of the street from me is one of Jersey City's old resorvoirs. It's not used for drinking water anymore and the city eventually plans on merging it into the park on the other side. In the meantime it has become something of a wildlife refuge. Which is nice except one of the newly settled critters was a bird of prey -- the consensus is possibly some kind of hawk or raven. Starting your morning commute under the eyes of a harbinger of death is very goth and I even learned to deal with the occasional piece of deconstructed rodent on my doorstep but nighttime was a big problem. For contrary to popular belief, ravens do not quoth "nevermore" but "KRRAAAA". Very loudly. Just as soon as you have drifted of to sleep. Eventually my sleep-deprived neighbors and I appealed to the NJ division of enviromental protection to get it removed but by the time they were ready to swing into action the bird had left for somewhere more congenial like Transylvania or Newark. Or here are some more complete wastes of time: I go the doctor for my annual physical. The insurance company codes it as Adult Onset Diabetes by accident. One day I opened the lid of my laptop and there's a "ping" sound and a piece of the hinge flies off. Apparently that also severed the connection to the screen and naturally the warranty had just expired so I had to spend the next month tethered to an external monitor until I could afford to buy a new one. Mix in all the usual social, political, family and work drama and you can see that it has been a very trying time for me. Dovecot I have managed to get some Debian work done. On Dovecot, my principal package, I have gotten tremendous support from Apollon Oikonomopolous who I belatedly welcome as a member of the Dovecot maintainer team. He has been particularly helpful in fixing our systemd support and cleaning out a lot of the old and invalid bugs. We're in pretty good shape for the freeze. Upstream has released an RC of 2.2.26 and hopefully the final version will be out in the next couple of days so we can include it in Stretch. We can always use more help with the package so let me know if you're interested. Debian-IN Most of the action has been going on without me but I've been lending support and sponsoring whenever I can. We have several new DDs and DMs but still no one north of the Vindhyas I'm afraid. Debian Perl Group gregoa did a ping of inactive maintainers and I regretfully had to admit to myself that I wasn't going to be of use anytime soon so I resigned. Perl remains my favorite language and I've actually been more involved in the meetings of my local Perlmongers group so hopefully I will be back again one day. And I still maintain the Perl modules I wrote myself. Debian-Axe-Murderers* May have gained a recruit. *Stricly speaking it should be called Debian-People-Who-Dont-Think-Faults-in-One-Moral-Domain-Such-As-For-Example-Axe-Murdering-Should-Leak-Into-Another-Moral-Domain-Such-As-For-Example-Debian but come on, that's just silly.

22 October 2016

Russell Coker: Another Broken Nexus 5

In late 2013 I bought a Nexus 5 for my wife [1]. It s a good phone and I generally have no complaints about the way it works. In the middle of 2016 I had to make a warranty claim when the original Nexus 5 stopped working [2]. Google s warranty support was ok, the call-back was good but unfortunately there was some confusion which delayed replacement. Once the confusion about the IMEI was resolved the warranty replacement method was to bill my credit card for a replacement phone and reverse the charge if/when they got the original phone back and found it to have a defect covered by warranty. This policy meant that I got a new phone sooner as they didn t need to get the old phone first. This is a huge benefit for defects that don t make the phone unusable as you will never be without a phone. Also if the user determines that the breakage was their fault they can just refrain from sending in the old phone. Today my wife s latest Nexus 5 developed a problem. It turned itself off and went into a reboot loop when connected to the charger. Also one of the clips on the rear case had popped out and other clips popped out when I pushed it back in. It appears (without opening the phone) that the battery may have grown larger (which is a common symptom of battery related problems). The phone is slightly less than 3 years old, so if I had got the extended warranty then I would have got a replacement. Now I m about to buy a Nexus 6P (because the Pixel is ridiculously expensive) which is $700 including postage. Kogan offers me a 3 year warranty for an extra $108. Obviously in retrospect spending an extra $100 would have been a benefit for the Nexus 5. But the first question is whether new phone going to have a probability greater than 1/7 of failing due to something other than user error in years 2 and 3? For an extended warranty to provide any benefit the phone has to have a problem that doesn t occur in the first year (or a problem in a replacement phone after the first phone was replaced). The phone also has to not be lost, stolen, or dropped in a pool by it s owner. While my wife and I have a good record of not losing or breaking phones the probability of it happening isn t zero. The Nexus 5 that just died can be replaced for 2/3 of the original price. The value of the old Nexus 5 to me is less than 2/3 of the original price as buying a newer better phone is the option I want. The value of an old phone to me decreases faster than the replacement cost because I don t want to buy an old phone. For an extended warranty to be a good deal for me I think it would have to cost significantly less than 1/10 of the purchase price due to the low probability of failure in that time period and the decreasing value of a replacement outdated phone. So even though my last choice to skip an extended warranty ended up not paying out I expect that overall I will be financially ahead if I keep self-insuring, and I m sure that I have already saved money by self-insuring all my previous devices.

[1] https://etbe.coker.com.au/2013/12/23/nexus-5/

[2] https://etbe.coker.com.au/2014/06/10/google-hardware-support/

Related posts:

The Nexus 5 The Nexus 5 is the latest Android phone to be...

Nexus 4 Ringke Fusion Case I ve been using Android phones for 2.5 years and...

Nexus 4 My wife has had a LG Nexus 4 for about...

Next.

Previous.